2023-04-19 20:53:31,150 DEBUG [main] hbase.HBaseTestingUtility(342): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b3568cac-f3a4-dbc0-683e-f8685e6f55fc 2023-04-19 20:53:31,165 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.TestAcidGuaranteesWithEagerPolicy timeout: 13 mins 2023-04-19 20:53:31,183 INFO [Time-limited test] hbase.HBaseTestingUtility(1068): Starting up minicluster with option: StartMiniClusterOption{numMasters=1, masterClass=null, numRegionServers=1, rsPorts=, rsClass=null, numDataNodes=1, dataNodeHosts=null, numZkServers=1, createRootDir=false, createWALDir=false} 2023-04-19 20:53:31,183 INFO [Time-limited test] hbase.HBaseZKTestingUtility(82): Created new mini-cluster data directory: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b3568cac-f3a4-dbc0-683e-f8685e6f55fc/cluster_dc7aee7d-e9b8-c554-2b32-38e4cf91128a, deleteOnExit=true 2023-04-19 20:53:31,183 INFO [Time-limited test] hbase.HBaseTestingUtility(1082): STARTING DFS 2023-04-19 20:53:31,184 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting test.cache.data to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b3568cac-f3a4-dbc0-683e-f8685e6f55fc/test.cache.data in system properties and HBase conf 2023-04-19 20:53:31,184 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.tmp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b3568cac-f3a4-dbc0-683e-f8685e6f55fc/hadoop.tmp.dir in system properties and HBase conf 2023-04-19 20:53:31,185 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.log.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b3568cac-f3a4-dbc0-683e-f8685e6f55fc/hadoop.log.dir in system properties and HBase conf 2023-04-19 20:53:31,185 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b3568cac-f3a4-dbc0-683e-f8685e6f55fc/mapreduce.cluster.local.dir in system properties and HBase conf 2023-04-19 20:53:31,186 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b3568cac-f3a4-dbc0-683e-f8685e6f55fc/mapreduce.cluster.temp.dir in system properties and HBase conf 2023-04-19 20:53:31,186 INFO [Time-limited test] hbase.HBaseTestingUtility(759): read short circuit is OFF 2023-04-19 20:53:31,300 WARN [Time-limited test] util.NativeCodeLoader(62): Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2023-04-19 20:53:31,723 DEBUG [Time-limited test] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering 2023-04-19 20:53:31,729 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-labels.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b3568cac-f3a4-dbc0-683e-f8685e6f55fc/yarn.node-labels.fs-store.root-dir in system properties and HBase conf 2023-04-19 20:53:31,729 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-attribute.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b3568cac-f3a4-dbc0-683e-f8685e6f55fc/yarn.node-attribute.fs-store.root-dir in system properties and HBase conf 2023-04-19 20:53:31,729 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.log-dirs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b3568cac-f3a4-dbc0-683e-f8685e6f55fc/yarn.nodemanager.log-dirs in system properties and HBase conf 2023-04-19 20:53:31,730 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b3568cac-f3a4-dbc0-683e-f8685e6f55fc/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-04-19 20:53:31,730 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.active-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b3568cac-f3a4-dbc0-683e-f8685e6f55fc/yarn.timeline-service.entity-group-fs-store.active-dir in system properties and HBase conf 2023-04-19 20:53:31,731 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.done-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b3568cac-f3a4-dbc0-683e-f8685e6f55fc/yarn.timeline-service.entity-group-fs-store.done-dir in system properties and HBase conf 2023-04-19 20:53:31,731 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b3568cac-f3a4-dbc0-683e-f8685e6f55fc/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-04-19 20:53:31,731 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b3568cac-f3a4-dbc0-683e-f8685e6f55fc/dfs.journalnode.edits.dir in system properties and HBase conf 2023-04-19 20:53:31,732 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.datanode.shared.file.descriptor.paths to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b3568cac-f3a4-dbc0-683e-f8685e6f55fc/dfs.datanode.shared.file.descriptor.paths in system properties and HBase conf 2023-04-19 20:53:31,732 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting nfs.dump.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b3568cac-f3a4-dbc0-683e-f8685e6f55fc/nfs.dump.dir in system properties and HBase conf 2023-04-19 20:53:31,732 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting java.io.tmpdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b3568cac-f3a4-dbc0-683e-f8685e6f55fc/java.io.tmpdir in system properties and HBase conf 2023-04-19 20:53:31,733 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b3568cac-f3a4-dbc0-683e-f8685e6f55fc/dfs.journalnode.edits.dir in system properties and HBase conf 2023-04-19 20:53:31,733 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.provided.aliasmap.inmemory.leveldb.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b3568cac-f3a4-dbc0-683e-f8685e6f55fc/dfs.provided.aliasmap.inmemory.leveldb.dir in system properties and HBase conf 2023-04-19 20:53:31,734 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting fs.s3a.committer.staging.tmp.path to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b3568cac-f3a4-dbc0-683e-f8685e6f55fc/fs.s3a.committer.staging.tmp.path in system properties and HBase conf Formatting using clusterid: testClusterID 2023-04-19 20:53:32,366 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-04-19 20:53:32,371 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-04-19 20:53:32,653 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties 2023-04-19 20:53:32,841 INFO [Time-limited test] log.Slf4jLog(67): Logging to org.slf4j.impl.Reload4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2023-04-19 20:53:32,860 WARN [Time-limited test] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-04-19 20:53:32,899 INFO [Time-limited test] log.Slf4jLog(67): jetty-6.1.26 2023-04-19 20:53:32,931 INFO [Time-limited test] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/hdfs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b3568cac-f3a4-dbc0-683e-f8685e6f55fc/java.io.tmpdir/Jetty_localhost_34393_hdfs____.82mimi/webapp 2023-04-19 20:53:33,071 INFO [Time-limited test] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:34393 2023-04-19 20:53:33,107 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-04-19 20:53:33,108 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-04-19 20:53:33,558 WARN [Listener at localhost/43431] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-04-19 20:53:33,692 WARN [Listener at localhost/43431] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-04-19 20:53:33,711 WARN [Listener at localhost/43431] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-04-19 20:53:33,719 INFO [Listener at localhost/43431] log.Slf4jLog(67): jetty-6.1.26 2023-04-19 20:53:33,726 INFO [Listener at localhost/43431] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b3568cac-f3a4-dbc0-683e-f8685e6f55fc/java.io.tmpdir/Jetty_localhost_41257_datanode____.l7lbky/webapp 2023-04-19 20:53:33,862 INFO [Listener at localhost/43431] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:41257 2023-04-19 20:53:34,265 WARN [Listener at localhost/34893] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-04-19 20:53:34,794 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x759727b152c17862: Processing first storage report for DS-f037c89d-927f-42a4-92da-f490f35d8621 from datanode db110760-0b31-424d-81dc-8e77f2c753f3 2023-04-19 20:53:34,796 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x759727b152c17862: from storage DS-f037c89d-927f-42a4-92da-f490f35d8621 node DatanodeRegistration(127.0.0.1:34473, datanodeUuid=db110760-0b31-424d-81dc-8e77f2c753f3, infoPort=44065, infoSecurePort=0, ipcPort=34893, storageInfo=lv=-57;cid=testClusterID;nsid=1541575988;c=1681937612442), blocks: 0, hasStaleStorage: true, processing time: 2 msecs, invalidatedBlocks: 0 2023-04-19 20:53:34,796 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x759727b152c17862: Processing first storage report for DS-7a474494-34c9-44d2-90fe-ca037376ef7f from datanode db110760-0b31-424d-81dc-8e77f2c753f3 2023-04-19 20:53:34,796 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x759727b152c17862: from storage DS-7a474494-34c9-44d2-90fe-ca037376ef7f node DatanodeRegistration(127.0.0.1:34473, datanodeUuid=db110760-0b31-424d-81dc-8e77f2c753f3, infoPort=44065, infoSecurePort=0, ipcPort=34893, storageInfo=lv=-57;cid=testClusterID;nsid=1541575988;c=1681937612442), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-04-19 20:53:34,835 DEBUG [Listener at localhost/34893] hbase.HBaseTestingUtility(649): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b3568cac-f3a4-dbc0-683e-f8685e6f55fc 2023-04-19 20:53:34,903 INFO [Listener at localhost/34893] zookeeper.MiniZooKeeperCluster(258): Started connectionTimeout=30000, dir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b3568cac-f3a4-dbc0-683e-f8685e6f55fc/cluster_dc7aee7d-e9b8-c554-2b32-38e4cf91128a/zookeeper_0, clientPort=58736, secureClientPort=-1, dataDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b3568cac-f3a4-dbc0-683e-f8685e6f55fc/cluster_dc7aee7d-e9b8-c554-2b32-38e4cf91128a/zookeeper_0/version-2, dataDirSize=424 dataLogDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b3568cac-f3a4-dbc0-683e-f8685e6f55fc/cluster_dc7aee7d-e9b8-c554-2b32-38e4cf91128a/zookeeper_0/version-2, dataLogSize=424 tickTime=2000, maxClientCnxns=300, minSessionTimeout=4000, maxSessionTimeout=40000, serverId=0 2023-04-19 20:53:34,917 INFO [Listener at localhost/34893] zookeeper.MiniZooKeeperCluster(283): Started MiniZooKeeperCluster and ran 'stat' on client port=58736 2023-04-19 20:53:34,924 INFO [Listener at localhost/34893] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-19 20:53:34,926 INFO [Listener at localhost/34893] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-19 20:53:35,586 INFO [Listener at localhost/34893] util.FSUtils(471): Created version file at hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49 with version=8 2023-04-19 20:53:35,586 INFO [Listener at localhost/34893] hbase.HBaseTestingUtility(1406): Setting hbase.fs.tmp.dir to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/hbase-staging 2023-04-19 20:53:35,906 INFO [Listener at localhost/34893] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-04-19 20:53:36,394 INFO [Listener at localhost/34893] client.ConnectionUtils(127): master/jenkins-hbase4:0 server-side Connection retries=45 2023-04-19 20:53:36,426 INFO [Listener at localhost/34893] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-19 20:53:36,427 INFO [Listener at localhost/34893] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-04-19 20:53:36,427 INFO [Listener at localhost/34893] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-04-19 20:53:36,427 INFO [Listener at localhost/34893] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-19 20:53:36,427 INFO [Listener at localhost/34893] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-04-19 20:53:36,585 INFO [Listener at localhost/34893] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-04-19 20:53:36,683 DEBUG [Listener at localhost/34893] util.ClassSize(228): Using Unsafe to estimate memory layout 2023-04-19 20:53:36,777 INFO [Listener at localhost/34893] ipc.NettyRpcServer(120): Bind to /172.31.14.131:37181 2023-04-19 20:53:36,787 INFO [Listener at localhost/34893] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-19 20:53:36,789 INFO [Listener at localhost/34893] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-19 20:53:36,809 INFO [Listener at localhost/34893] zookeeper.RecoverableZooKeeper(93): Process identifier=master:37181 connecting to ZooKeeper ensemble=127.0.0.1:58736 2023-04-19 20:53:36,851 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:371810x0, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-04-19 20:53:36,853 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): master:37181-0x1019e3331ea0000 connected 2023-04-19 20:53:36,882 DEBUG [Listener at localhost/34893] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-04-19 20:53:36,883 DEBUG [Listener at localhost/34893] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-19 20:53:36,886 DEBUG [Listener at localhost/34893] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-04-19 20:53:36,893 DEBUG [Listener at localhost/34893] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=37181 2023-04-19 20:53:36,894 DEBUG [Listener at localhost/34893] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=37181 2023-04-19 20:53:36,894 DEBUG [Listener at localhost/34893] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=37181 2023-04-19 20:53:36,895 DEBUG [Listener at localhost/34893] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=37181 2023-04-19 20:53:36,895 DEBUG [Listener at localhost/34893] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=37181 2023-04-19 20:53:36,901 INFO [Listener at localhost/34893] master.HMaster(444): hbase.rootdir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49, hbase.cluster.distributed=false 2023-04-19 20:53:36,972 INFO [Listener at localhost/34893] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=45 2023-04-19 20:53:36,972 INFO [Listener at localhost/34893] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-19 20:53:36,973 INFO [Listener at localhost/34893] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-04-19 20:53:36,973 INFO [Listener at localhost/34893] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-04-19 20:53:36,973 INFO [Listener at localhost/34893] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-19 20:53:36,973 INFO [Listener at localhost/34893] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-04-19 20:53:36,978 INFO [Listener at localhost/34893] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-04-19 20:53:36,981 INFO [Listener at localhost/34893] ipc.NettyRpcServer(120): Bind to /172.31.14.131:35247 2023-04-19 20:53:36,983 INFO [Listener at localhost/34893] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-04-19 20:53:36,989 DEBUG [Listener at localhost/34893] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-04-19 20:53:36,990 INFO [Listener at localhost/34893] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-19 20:53:36,992 INFO [Listener at localhost/34893] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-19 20:53:36,994 INFO [Listener at localhost/34893] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:35247 connecting to ZooKeeper ensemble=127.0.0.1:58736 2023-04-19 20:53:36,998 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:352470x0, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-04-19 20:53:36,999 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:35247-0x1019e3331ea0001 connected 2023-04-19 20:53:36,999 DEBUG [Listener at localhost/34893] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-04-19 20:53:37,000 DEBUG [Listener at localhost/34893] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-19 20:53:37,001 DEBUG [Listener at localhost/34893] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-04-19 20:53:37,001 DEBUG [Listener at localhost/34893] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=35247 2023-04-19 20:53:37,002 DEBUG [Listener at localhost/34893] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=35247 2023-04-19 20:53:37,004 DEBUG [Listener at localhost/34893] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=35247 2023-04-19 20:53:37,005 DEBUG [Listener at localhost/34893] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=35247 2023-04-19 20:53:37,006 DEBUG [Listener at localhost/34893] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=35247 2023-04-19 20:53:37,008 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2168): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase4.apache.org,37181,1681937615733 2023-04-19 20:53:37,018 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-04-19 20:53:37,019 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase4.apache.org,37181,1681937615733 2023-04-19 20:53:37,038 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-04-19 20:53:37,038 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-04-19 20:53:37,038 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-19 20:53:37,040 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-04-19 20:53:37,041 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(227): Deleting ZNode for /hbase/backup-masters/jenkins-hbase4.apache.org,37181,1681937615733 from backup master directory 2023-04-19 20:53:37,041 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-04-19 20:53:37,045 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase4.apache.org,37181,1681937615733 2023-04-19 20:53:37,046 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-04-19 20:53:37,046 WARN [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-04-19 20:53:37,046 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(237): Registered as active master=jenkins-hbase4.apache.org,37181,1681937615733 2023-04-19 20:53:37,049 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating data MemStoreChunkPool with chunk size 2 MB, max count 352, initial count 0 2023-04-19 20:53:37,051 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating index MemStoreChunkPool with chunk size 204.80 KB, max count 391, initial count 0 2023-04-19 20:53:37,138 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] util.FSUtils(620): Created cluster ID file at hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/hbase.id with ID: 8a79bd36-1895-4ca7-82f6-e01ee84ee126 2023-04-19 20:53:37,181 INFO [master/jenkins-hbase4:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-19 20:53:37,199 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-19 20:53:37,241 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x2c57173e to 127.0.0.1:58736 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-19 20:53:37,277 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6b7c9720, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-19 20:53:37,303 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-19 20:53:37,306 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-04-19 20:53:37,326 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(264): ClientProtocol::create wrong number of arguments, should be hadoop 3.2 or below 2023-04-19 20:53:37,326 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(270): ClientProtocol::create wrong number of arguments, should be hadoop 2.x 2023-04-19 20:53:37,328 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(279): can not find SHOULD_REPLICATE flag, should be hadoop 2.x java.lang.IllegalArgumentException: No enum constant org.apache.hadoop.fs.CreateFlag.SHOULD_REPLICATE at java.lang.Enum.valueOf(Enum.java:238) at org.apache.hadoop.fs.CreateFlag.valueOf(CreateFlag.java:63) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.loadShouldReplicateFlag(FanOutOneBlockAsyncDFSOutputHelper.java:277) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.(FanOutOneBlockAsyncDFSOutputHelper.java:304) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:139) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-04-19 20:53:37,332 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(243): No decryptEncryptedDataEncryptionKey method in DFSClient, should be hadoop version with HDFS-12396 java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(org.apache.hadoop.fs.FileEncryptionInfo) at java.lang.Class.getDeclaredMethod(Class.java:2130) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelperWithoutHDFS12396(FanOutOneBlockAsyncDFSOutputSaslHelper.java:182) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelper(FanOutOneBlockAsyncDFSOutputSaslHelper.java:241) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.(FanOutOneBlockAsyncDFSOutputSaslHelper.java:252) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:140) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-04-19 20:53:37,333 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-04-19 20:53:37,370 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7693): Creating {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, under table dir hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/MasterData/data/master/store-tmp 2023-04-19 20:53:37,399 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-19 20:53:37,399 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-04-19 20:53:37,399 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-19 20:53:37,399 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-19 20:53:37,400 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-04-19 20:53:37,400 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-19 20:53:37,400 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-19 20:53:37,400 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-04-19 20:53:37,401 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/MasterData/WALs/jenkins-hbase4.apache.org,37181,1681937615733 2023-04-19 20:53:37,423 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C37181%2C1681937615733, suffix=, logDir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/MasterData/WALs/jenkins-hbase4.apache.org,37181,1681937615733, archiveDir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/MasterData/oldWALs, maxLogs=10 2023-04-19 20:53:37,481 DEBUG [RS-EventLoopGroup-3-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:34473,DS-f037c89d-927f-42a4-92da-f490f35d8621,DISK] 2023-04-19 20:53:37,489 DEBUG [RS-EventLoopGroup-3-2] asyncfs.ProtobufDecoder(123): Hadoop 3.2 and below use unshaded protobuf. java.lang.ClassNotFoundException: org.apache.hadoop.thirdparty.protobuf.MessageLite at java.net.URLClassLoader.findClass(URLClassLoader.java:387) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.io.asyncfs.ProtobufDecoder.(ProtobufDecoder.java:118) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.processWriteBlockResponse(FanOutOneBlockAsyncDFSOutputHelper.java:340) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$100(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$4.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:424) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:185) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.initialize(FanOutOneBlockAsyncDFSOutputHelper.java:418) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$300(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:476) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:471) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:105) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.fulfillConnectPromise(AbstractEpollChannel.java:653) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:691) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-19 20:53:37,556 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/MasterData/WALs/jenkins-hbase4.apache.org,37181,1681937615733/jenkins-hbase4.apache.org%2C37181%2C1681937615733.1681937617435 2023-04-19 20:53:37,556 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:34473,DS-f037c89d-927f-42a4-92da-f490f35d8621,DISK]] 2023-04-19 20:53:37,557 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7854): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-04-19 20:53:37,557 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-19 20:53:37,560 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7894): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-04-19 20:53:37,561 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7897): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-04-19 20:53:37,623 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-04-19 20:53:37,630 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-04-19 20:53:37,654 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-04-19 20:53:37,666 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-19 20:53:37,672 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-04-19 20:53:37,674 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-04-19 20:53:37,691 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1055): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-04-19 20:53:37,696 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-19 20:53:37,697 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1072): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=65789231, jitterRate=-0.019664064049720764}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-04-19 20:53:37,697 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(965): Region open journal for 1595e783b53d99cd5eef43b6debb2682: 2023-04-19 20:53:37,700 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-04-19 20:53:37,719 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=5 2023-04-19 20:53:37,719 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 5 core workers (bigger of cpus/4 or 16) with max (burst) worker count=50 2023-04-19 20:53:37,722 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-04-19 20:53:37,724 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 1 msec 2023-04-19 20:53:37,758 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 33 msec 2023-04-19 20:53:37,758 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(95): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-04-19 20:53:37,786 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [] 2023-04-19 20:53:37,792 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2023-04-19 20:53:37,821 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-04-19 20:53:37,826 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-04-19 20:53:37,828 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-04-19 20:53:37,834 INFO [master/jenkins-hbase4:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-04-19 20:53:37,839 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-04-19 20:53:37,843 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-19 20:53:37,845 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-04-19 20:53:37,845 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-04-19 20:53:37,857 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-04-19 20:53:37,861 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-04-19 20:53:37,861 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-04-19 20:53:37,861 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-19 20:53:37,862 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(744): Active/primary master=jenkins-hbase4.apache.org,37181,1681937615733, sessionid=0x1019e3331ea0000, setting cluster-up flag (Was=false) 2023-04-19 20:53:37,881 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-19 20:53:37,888 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-04-19 20:53:37,890 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,37181,1681937615733 2023-04-19 20:53:37,896 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-19 20:53:37,902 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-04-19 20:53:37,903 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,37181,1681937615733 2023-04-19 20:53:37,907 WARN [master/jenkins-hbase4:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/.hbase-snapshot/.tmp 2023-04-19 20:53:37,913 INFO [RS:0;jenkins-hbase4:35247] regionserver.HRegionServer(952): ClusterId : 8a79bd36-1895-4ca7-82f6-e01ee84ee126 2023-04-19 20:53:37,919 DEBUG [RS:0;jenkins-hbase4:35247] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-04-19 20:53:37,926 DEBUG [RS:0;jenkins-hbase4:35247] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-04-19 20:53:37,926 DEBUG [RS:0;jenkins-hbase4:35247] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-04-19 20:53:37,934 DEBUG [RS:0;jenkins-hbase4:35247] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-04-19 20:53:37,936 DEBUG [RS:0;jenkins-hbase4:35247] zookeeper.ReadOnlyZKClient(139): Connect 0x735e461f to 127.0.0.1:58736 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-19 20:53:37,947 DEBUG [RS:0;jenkins-hbase4:35247] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7d99b604, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-19 20:53:37,948 DEBUG [RS:0;jenkins-hbase4:35247] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@79a62d6f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-04-19 20:53:37,977 DEBUG [RS:0;jenkins-hbase4:35247] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:0;jenkins-hbase4:35247 2023-04-19 20:53:37,982 INFO [RS:0;jenkins-hbase4:35247] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-04-19 20:53:37,982 INFO [RS:0;jenkins-hbase4:35247] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-04-19 20:53:37,982 DEBUG [RS:0;jenkins-hbase4:35247] regionserver.HRegionServer(1023): About to register with Master. 2023-04-19 20:53:37,986 INFO [RS:0;jenkins-hbase4:35247] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,37181,1681937615733 with isa=jenkins-hbase4.apache.org/172.31.14.131:35247, startcode=1681937616971 2023-04-19 20:53:38,004 DEBUG [RS:0;jenkins-hbase4:35247] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-04-19 20:53:38,069 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT; InitMetaProcedure table=hbase:meta 2023-04-19 20:53:38,072 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52161, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.0 (auth:SIMPLE), service=RegionServerStatusService 2023-04-19 20:53:38,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerStartup(MasterRpcServices.java:577) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15952) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:38,082 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-04-19 20:53:38,082 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-04-19 20:53:38,082 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-04-19 20:53:38,082 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-04-19 20:53:38,082 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase4:0, corePoolSize=10, maxPoolSize=10 2023-04-19 20:53:38,082 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-19 20:53:38,082 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-04-19 20:53:38,083 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-19 20:53:38,086 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1681937648086 2023-04-19 20:53:38,090 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-04-19 20:53:38,098 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta 2023-04-19 20:53:38,099 INFO [PEWorker-1] procedure.InitMetaProcedure(71): BOOTSTRAP: creating hbase:meta region 2023-04-19 20:53:38,104 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-04-19 20:53:38,106 INFO [PEWorker-1] util.FSTableDescriptors(128): Creating new hbase:meta table descriptor 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-04-19 20:53:38,115 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-04-19 20:53:38,116 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-04-19 20:53:38,116 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-04-19 20:53:38,116 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-04-19 20:53:38,119 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-04-19 20:53:38,123 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-04-19 20:53:38,124 DEBUG [RS:0;jenkins-hbase4:35247] regionserver.HRegionServer(2829): Master is not running yet 2023-04-19 20:53:38,125 WARN [RS:0;jenkins-hbase4:35247] regionserver.HRegionServer(1031): reportForDuty failed; sleeping 100 ms and then retrying. 2023-04-19 20:53:38,125 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-04-19 20:53:38,126 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-04-19 20:53:38,131 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-04-19 20:53:38,131 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-04-19 20:53:38,134 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1681937618133,5,FailOnTimeoutGroup] 2023-04-19 20:53:38,134 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1681937618134,5,FailOnTimeoutGroup] 2023-04-19 20:53:38,134 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-04-19 20:53:38,134 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1461): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-04-19 20:53:38,136 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-04-19 20:53:38,136 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-04-19 20:53:38,146 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-04-19 20:53:38,147 INFO [PEWorker-1] util.FSTableDescriptors(135): Updated hbase:meta table descriptor to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-04-19 20:53:38,147 INFO [PEWorker-1] regionserver.HRegion(7675): creating {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49 2023-04-19 20:53:38,226 INFO [RS:0;jenkins-hbase4:35247] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,37181,1681937615733 with isa=jenkins-hbase4.apache.org/172.31.14.131:35247, startcode=1681937616971 2023-04-19 20:53:38,232 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:38,239 DEBUG [RS:0;jenkins-hbase4:35247] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49 2023-04-19 20:53:38,239 DEBUG [RS:0;jenkins-hbase4:35247] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:43431 2023-04-19 20:53:38,239 DEBUG [RS:0;jenkins-hbase4:35247] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-04-19 20:53:38,246 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-04-19 20:53:38,247 DEBUG [RS:0;jenkins-hbase4:35247] zookeeper.ZKUtil(162): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:38,247 WARN [RS:0;jenkins-hbase4:35247] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-04-19 20:53:38,247 INFO [RS:0;jenkins-hbase4:35247] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-04-19 20:53:38,248 DEBUG [RS:0;jenkins-hbase4:35247] regionserver.HRegionServer(1947): logDir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/WALs/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:38,248 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,35247,1681937616971] 2023-04-19 20:53:38,258 DEBUG [RS:0;jenkins-hbase4:35247] zookeeper.ZKUtil(162): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:38,268 DEBUG [RS:0;jenkins-hbase4:35247] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-04-19 20:53:38,277 INFO [RS:0;jenkins-hbase4:35247] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-04-19 20:53:38,297 INFO [RS:0;jenkins-hbase4:35247] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-04-19 20:53:38,301 INFO [RS:0;jenkins-hbase4:35247] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-04-19 20:53:38,302 INFO [RS:0;jenkins-hbase4:35247] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-04-19 20:53:38,303 INFO [RS:0;jenkins-hbase4:35247] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-04-19 20:53:38,309 INFO [RS:0;jenkins-hbase4:35247] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-04-19 20:53:38,310 DEBUG [RS:0;jenkins-hbase4:35247] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-19 20:53:38,310 DEBUG [RS:0;jenkins-hbase4:35247] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-19 20:53:38,310 DEBUG [RS:0;jenkins-hbase4:35247] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-19 20:53:38,310 DEBUG [RS:0;jenkins-hbase4:35247] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-19 20:53:38,310 DEBUG [RS:0;jenkins-hbase4:35247] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-19 20:53:38,310 DEBUG [RS:0;jenkins-hbase4:35247] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-04-19 20:53:38,310 DEBUG [RS:0;jenkins-hbase4:35247] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-19 20:53:38,311 DEBUG [RS:0;jenkins-hbase4:35247] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-19 20:53:38,311 DEBUG [RS:0;jenkins-hbase4:35247] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-19 20:53:38,311 DEBUG [RS:0;jenkins-hbase4:35247] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-19 20:53:38,315 INFO [RS:0;jenkins-hbase4:35247] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-04-19 20:53:38,315 INFO [RS:0;jenkins-hbase4:35247] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-04-19 20:53:38,315 INFO [RS:0;jenkins-hbase4:35247] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-04-19 20:53:38,334 INFO [RS:0;jenkins-hbase4:35247] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-04-19 20:53:38,337 INFO [RS:0;jenkins-hbase4:35247] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,35247,1681937616971-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-04-19 20:53:38,356 INFO [RS:0;jenkins-hbase4:35247] regionserver.Replication(203): jenkins-hbase4.apache.org,35247,1681937616971 started 2023-04-19 20:53:38,356 INFO [RS:0;jenkins-hbase4:35247] regionserver.HRegionServer(1638): Serving as jenkins-hbase4.apache.org,35247,1681937616971, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:35247, sessionid=0x1019e3331ea0001 2023-04-19 20:53:38,356 DEBUG [RS:0;jenkins-hbase4:35247] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-04-19 20:53:38,356 DEBUG [RS:0;jenkins-hbase4:35247] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:38,356 DEBUG [RS:0;jenkins-hbase4:35247] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,35247,1681937616971' 2023-04-19 20:53:38,356 DEBUG [RS:0;jenkins-hbase4:35247] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:53:38,357 DEBUG [RS:0;jenkins-hbase4:35247] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:53:38,357 DEBUG [RS:0;jenkins-hbase4:35247] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-04-19 20:53:38,358 DEBUG [RS:0;jenkins-hbase4:35247] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-04-19 20:53:38,358 DEBUG [RS:0;jenkins-hbase4:35247] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:38,358 DEBUG [RS:0;jenkins-hbase4:35247] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,35247,1681937616971' 2023-04-19 20:53:38,358 DEBUG [RS:0;jenkins-hbase4:35247] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-04-19 20:53:38,358 DEBUG [RS:0;jenkins-hbase4:35247] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-04-19 20:53:38,359 DEBUG [RS:0;jenkins-hbase4:35247] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-04-19 20:53:38,359 INFO [RS:0;jenkins-hbase4:35247] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-04-19 20:53:38,359 INFO [RS:0;jenkins-hbase4:35247] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-04-19 20:53:38,472 INFO [RS:0;jenkins-hbase4:35247] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C35247%2C1681937616971, suffix=, logDir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/WALs/jenkins-hbase4.apache.org,35247,1681937616971, archiveDir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/oldWALs, maxLogs=32 2023-04-19 20:53:38,497 DEBUG [RS-EventLoopGroup-3-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:34473,DS-f037c89d-927f-42a4-92da-f490f35d8621,DISK] 2023-04-19 20:53:38,501 INFO [RS:0;jenkins-hbase4:35247] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/WALs/jenkins-hbase4.apache.org,35247,1681937616971/jenkins-hbase4.apache.org%2C35247%2C1681937616971.1681937618475 2023-04-19 20:53:38,502 DEBUG [RS:0;jenkins-hbase4:35247] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:34473,DS-f037c89d-927f-42a4-92da-f490f35d8621,DISK]] 2023-04-19 20:53:38,564 DEBUG [PEWorker-1] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-19 20:53:38,567 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-04-19 20:53:38,572 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/hbase/meta/1588230740/info 2023-04-19 20:53:38,573 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-04-19 20:53:38,575 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-19 20:53:38,575 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-04-19 20:53:38,578 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/hbase/meta/1588230740/rep_barrier 2023-04-19 20:53:38,579 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-04-19 20:53:38,580 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-19 20:53:38,580 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-04-19 20:53:38,582 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/hbase/meta/1588230740/table 2023-04-19 20:53:38,583 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-04-19 20:53:38,584 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-19 20:53:38,586 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/hbase/meta/1588230740 2023-04-19 20:53:38,587 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/hbase/meta/1588230740 2023-04-19 20:53:38,591 DEBUG [PEWorker-1] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-19 20:53:38,594 DEBUG [PEWorker-1] regionserver.HRegion(1055): writing seq id for 1588230740 2023-04-19 20:53:38,598 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-19 20:53:38,599 INFO [PEWorker-1] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59774577, jitterRate=-0.10928939282894135}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-19 20:53:38,600 DEBUG [PEWorker-1] regionserver.HRegion(965): Region open journal for 1588230740: 2023-04-19 20:53:38,600 DEBUG [PEWorker-1] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-04-19 20:53:38,600 INFO [PEWorker-1] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-04-19 20:53:38,600 DEBUG [PEWorker-1] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-04-19 20:53:38,600 DEBUG [PEWorker-1] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-04-19 20:53:38,600 DEBUG [PEWorker-1] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-04-19 20:53:38,602 INFO [PEWorker-1] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-04-19 20:53:38,602 DEBUG [PEWorker-1] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-04-19 20:53:38,607 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, locked=true; InitMetaProcedure table=hbase:meta 2023-04-19 20:53:38,607 INFO [PEWorker-1] procedure.InitMetaProcedure(103): Going to assign meta 2023-04-19 20:53:38,617 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-04-19 20:53:38,631 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-04-19 20:53:38,634 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-04-19 20:53:38,786 DEBUG [jenkins-hbase4:37181] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-04-19 20:53:38,790 INFO [PEWorker-3] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,35247,1681937616971, state=OPENING 2023-04-19 20:53:38,799 DEBUG [PEWorker-3] zookeeper.MetaTableLocator(240): hbase:meta region location doesn't exist, create it 2023-04-19 20:53:38,801 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-19 20:53:38,802 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-04-19 20:53:38,806 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,35247,1681937616971}] 2023-04-19 20:53:39,001 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:39,004 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-04-19 20:53:39,007 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:42288, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-04-19 20:53:39,021 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open hbase:meta,,1.1588230740 2023-04-19 20:53:39,021 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-04-19 20:53:39,025 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C35247%2C1681937616971.meta, suffix=.meta, logDir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/WALs/jenkins-hbase4.apache.org,35247,1681937616971, archiveDir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/oldWALs, maxLogs=32 2023-04-19 20:53:39,043 DEBUG [RS-EventLoopGroup-3-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:34473,DS-f037c89d-927f-42a4-92da-f490f35d8621,DISK] 2023-04-19 20:53:39,047 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/WALs/jenkins-hbase4.apache.org,35247,1681937616971/jenkins-hbase4.apache.org%2C35247%2C1681937616971.meta.1681937619027.meta 2023-04-19 20:53:39,048 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:34473,DS-f037c89d-927f-42a4-92da-f490f35d8621,DISK]] 2023-04-19 20:53:39,048 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-04-19 20:53:39,050 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-04-19 20:53:39,067 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-04-19 20:53:39,072 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-04-19 20:53:39,078 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-04-19 20:53:39,078 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-19 20:53:39,078 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-04-19 20:53:39,078 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-04-19 20:53:39,081 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-04-19 20:53:39,083 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/hbase/meta/1588230740/info 2023-04-19 20:53:39,083 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/hbase/meta/1588230740/info 2023-04-19 20:53:39,084 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-04-19 20:53:39,085 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-19 20:53:39,085 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-04-19 20:53:39,087 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/hbase/meta/1588230740/rep_barrier 2023-04-19 20:53:39,087 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/hbase/meta/1588230740/rep_barrier 2023-04-19 20:53:39,087 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-04-19 20:53:39,088 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-19 20:53:39,088 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-04-19 20:53:39,090 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/hbase/meta/1588230740/table 2023-04-19 20:53:39,090 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/hbase/meta/1588230740/table 2023-04-19 20:53:39,091 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-04-19 20:53:39,091 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-19 20:53:39,093 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/hbase/meta/1588230740 2023-04-19 20:53:39,096 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/hbase/meta/1588230740 2023-04-19 20:53:39,099 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-19 20:53:39,102 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-04-19 20:53:39,103 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61722142, jitterRate=-0.08026841282844543}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-19 20:53:39,104 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-04-19 20:53:39,114 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:meta,,1.1588230740, pid=3, masterSystemTime=1681937618992 2023-04-19 20:53:39,135 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:meta,,1.1588230740 2023-04-19 20:53:39,135 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened hbase:meta,,1.1588230740 2023-04-19 20:53:39,136 INFO [PEWorker-5] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,35247,1681937616971, state=OPEN 2023-04-19 20:53:39,139 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-04-19 20:53:39,139 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-04-19 20:53:39,144 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=3, resume processing ppid=2 2023-04-19 20:53:39,144 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=3, ppid=2, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,35247,1681937616971 in 333 msec 2023-04-19 20:53:39,151 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=2, resume processing ppid=1 2023-04-19 20:53:39,151 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=2, ppid=1, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 530 msec 2023-04-19 20:53:39,157 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 1.1760 sec 2023-04-19 20:53:39,158 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(953): Wait for region servers to report in: status=null, state=RUNNING, startTime=1681937619157, completionTime=-1 2023-04-19 20:53:39,158 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=1; waited=0ms, expected min=1 server(s), max=1 server(s), master is running 2023-04-19 20:53:39,158 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1517): Joining cluster... 2023-04-19 20:53:39,226 DEBUG [hconnection-0x2a4b69b2-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-19 20:53:39,231 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:42302, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-19 20:53:39,251 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1529): Number of RegionServers=1 2023-04-19 20:53:39,251 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1681937679251 2023-04-19 20:53:39,251 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1681937739251 2023-04-19 20:53:39,252 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1536): Joined the cluster in 93 msec 2023-04-19 20:53:39,278 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,37181,1681937615733-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-04-19 20:53:39,279 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,37181,1681937615733-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-04-19 20:53:39,279 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,37181,1681937615733-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-04-19 20:53:39,281 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase4:37181, period=300000, unit=MILLISECONDS is enabled. 2023-04-19 20:53:39,281 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-04-19 20:53:39,288 DEBUG [master/jenkins-hbase4:0.Chore.1] janitor.CatalogJanitor(175): 2023-04-19 20:53:39,299 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.TableNamespaceManager(92): Namespace table not found. Creating... 2023-04-19 20:53:39,300 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2148): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-04-19 20:53:39,311 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2023-04-19 20:53:39,314 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_PRE_OPERATION 2023-04-19 20:53:39,317 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-19 20:53:39,340 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/.tmp/data/hbase/namespace/e22e0fa16aeb37f0afb0679ab2e27008 2023-04-19 20:53:39,344 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/.tmp/data/hbase/namespace/e22e0fa16aeb37f0afb0679ab2e27008 empty. 2023-04-19 20:53:39,344 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/.tmp/data/hbase/namespace/e22e0fa16aeb37f0afb0679ab2e27008 2023-04-19 20:53:39,345 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived hbase:namespace regions 2023-04-19 20:53:39,770 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2023-04-19 20:53:39,773 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(7675): creating {ENCODED => e22e0fa16aeb37f0afb0679ab2e27008, NAME => 'hbase:namespace,,1681937619300.e22e0fa16aeb37f0afb0679ab2e27008.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/.tmp 2023-04-19 20:53:39,787 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1681937619300.e22e0fa16aeb37f0afb0679ab2e27008.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-19 20:53:39,787 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1604): Closing e22e0fa16aeb37f0afb0679ab2e27008, disabling compactions & flushes 2023-04-19 20:53:39,787 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1681937619300.e22e0fa16aeb37f0afb0679ab2e27008. 2023-04-19 20:53:39,787 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1681937619300.e22e0fa16aeb37f0afb0679ab2e27008. 2023-04-19 20:53:39,787 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1681937619300.e22e0fa16aeb37f0afb0679ab2e27008. after waiting 0 ms 2023-04-19 20:53:39,787 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1681937619300.e22e0fa16aeb37f0afb0679ab2e27008. 2023-04-19 20:53:39,788 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1838): Closed hbase:namespace,,1681937619300.e22e0fa16aeb37f0afb0679ab2e27008. 2023-04-19 20:53:39,788 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1558): Region close journal for e22e0fa16aeb37f0afb0679ab2e27008: 2023-04-19 20:53:39,792 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ADD_TO_META 2023-04-19 20:53:39,807 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:namespace,,1681937619300.e22e0fa16aeb37f0afb0679ab2e27008.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1681937619795"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681937619795"}]},"ts":"1681937619795"} 2023-04-19 20:53:39,834 INFO [PEWorker-3] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-19 20:53:39,836 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-19 20:53:39,840 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681937619836"}]},"ts":"1681937619836"} 2023-04-19 20:53:39,845 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2023-04-19 20:53:39,855 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=e22e0fa16aeb37f0afb0679ab2e27008, ASSIGN}] 2023-04-19 20:53:39,858 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=e22e0fa16aeb37f0afb0679ab2e27008, ASSIGN 2023-04-19 20:53:39,859 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=e22e0fa16aeb37f0afb0679ab2e27008, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,35247,1681937616971; forceNewPlan=false, retain=false 2023-04-19 20:53:40,010 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=e22e0fa16aeb37f0afb0679ab2e27008, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:40,010 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1681937619300.e22e0fa16aeb37f0afb0679ab2e27008.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1681937620010"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681937620010"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681937620010"}]},"ts":"1681937620010"} 2023-04-19 20:53:40,017 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=6, ppid=5, state=RUNNABLE; OpenRegionProcedure e22e0fa16aeb37f0afb0679ab2e27008, server=jenkins-hbase4.apache.org,35247,1681937616971}] 2023-04-19 20:53:40,178 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open hbase:namespace,,1681937619300.e22e0fa16aeb37f0afb0679ab2e27008. 2023-04-19 20:53:40,178 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => e22e0fa16aeb37f0afb0679ab2e27008, NAME => 'hbase:namespace,,1681937619300.e22e0fa16aeb37f0afb0679ab2e27008.', STARTKEY => '', ENDKEY => ''} 2023-04-19 20:53:40,179 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace e22e0fa16aeb37f0afb0679ab2e27008 2023-04-19 20:53:40,180 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1681937619300.e22e0fa16aeb37f0afb0679ab2e27008.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-19 20:53:40,180 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for e22e0fa16aeb37f0afb0679ab2e27008 2023-04-19 20:53:40,180 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for e22e0fa16aeb37f0afb0679ab2e27008 2023-04-19 20:53:40,182 INFO [StoreOpener-e22e0fa16aeb37f0afb0679ab2e27008-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region e22e0fa16aeb37f0afb0679ab2e27008 2023-04-19 20:53:40,185 DEBUG [StoreOpener-e22e0fa16aeb37f0afb0679ab2e27008-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/hbase/namespace/e22e0fa16aeb37f0afb0679ab2e27008/info 2023-04-19 20:53:40,185 DEBUG [StoreOpener-e22e0fa16aeb37f0afb0679ab2e27008-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/hbase/namespace/e22e0fa16aeb37f0afb0679ab2e27008/info 2023-04-19 20:53:40,186 INFO [StoreOpener-e22e0fa16aeb37f0afb0679ab2e27008-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e22e0fa16aeb37f0afb0679ab2e27008 columnFamilyName info 2023-04-19 20:53:40,186 INFO [StoreOpener-e22e0fa16aeb37f0afb0679ab2e27008-1] regionserver.HStore(310): Store=e22e0fa16aeb37f0afb0679ab2e27008/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-19 20:53:40,188 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/hbase/namespace/e22e0fa16aeb37f0afb0679ab2e27008 2023-04-19 20:53:40,189 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/hbase/namespace/e22e0fa16aeb37f0afb0679ab2e27008 2023-04-19 20:53:40,194 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for e22e0fa16aeb37f0afb0679ab2e27008 2023-04-19 20:53:40,198 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/hbase/namespace/e22e0fa16aeb37f0afb0679ab2e27008/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-19 20:53:40,200 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened e22e0fa16aeb37f0afb0679ab2e27008; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=69028676, jitterRate=0.028607428073883057}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-04-19 20:53:40,200 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for e22e0fa16aeb37f0afb0679ab2e27008: 2023-04-19 20:53:40,202 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:namespace,,1681937619300.e22e0fa16aeb37f0afb0679ab2e27008., pid=6, masterSystemTime=1681937620171 2023-04-19 20:53:40,205 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:namespace,,1681937619300.e22e0fa16aeb37f0afb0679ab2e27008. 2023-04-19 20:53:40,205 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened hbase:namespace,,1681937619300.e22e0fa16aeb37f0afb0679ab2e27008. 2023-04-19 20:53:40,207 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=e22e0fa16aeb37f0afb0679ab2e27008, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:40,208 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1681937619300.e22e0fa16aeb37f0afb0679ab2e27008.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1681937620206"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681937620206"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681937620206"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681937620206"}]},"ts":"1681937620206"} 2023-04-19 20:53:40,215 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=6, resume processing ppid=5 2023-04-19 20:53:40,215 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=6, ppid=5, state=SUCCESS; OpenRegionProcedure e22e0fa16aeb37f0afb0679ab2e27008, server=jenkins-hbase4.apache.org,35247,1681937616971 in 194 msec 2023-04-19 20:53:40,219 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=5, resume processing ppid=4 2023-04-19 20:53:40,220 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=5, ppid=4, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=e22e0fa16aeb37f0afb0679ab2e27008, ASSIGN in 360 msec 2023-04-19 20:53:40,221 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-19 20:53:40,221 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681937620221"}]},"ts":"1681937620221"} 2023-04-19 20:53:40,224 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2023-04-19 20:53:40,228 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_POST_OPERATION 2023-04-19 20:53:40,231 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 927 msec 2023-04-19 20:53:40,315 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2023-04-19 20:53:40,316 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2023-04-19 20:53:40,316 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-19 20:53:40,357 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=7, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2023-04-19 20:53:40,410 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-04-19 20:53:40,416 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 67 msec 2023-04-19 20:53:40,424 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=8, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2023-04-19 20:53:40,436 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-04-19 20:53:40,441 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 16 msec 2023-04-19 20:53:40,451 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-04-19 20:53:40,454 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-04-19 20:53:40,454 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1083): Master has completed initialization 3.407sec 2023-04-19 20:53:40,456 INFO [master/jenkins-hbase4:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-04-19 20:53:40,457 INFO [master/jenkins-hbase4:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-04-19 20:53:40,457 INFO [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-04-19 20:53:40,459 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,37181,1681937615733-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-04-19 20:53:40,460 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,37181,1681937615733-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-04-19 20:53:40,477 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1175): Balancer post startup initialization complete, took 0 seconds 2023-04-19 20:53:40,519 DEBUG [Listener at localhost/34893] zookeeper.ReadOnlyZKClient(139): Connect 0x461317a6 to 127.0.0.1:58736 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-19 20:53:40,524 DEBUG [Listener at localhost/34893] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@39efd3b0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-19 20:53:40,538 DEBUG [hconnection-0xb966214-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-19 20:53:40,550 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:42306, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-19 20:53:40,560 INFO [Listener at localhost/34893] hbase.HBaseTestingUtility(1145): Minicluster is up; activeMaster=jenkins-hbase4.apache.org,37181,1681937615733 2023-04-19 20:53:40,581 INFO [Listener at localhost/34893] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMixedAtomicity Thread=206, OpenFileDescriptor=470, MaxFileDescriptor=60000, SystemLoadAverage=337, ProcessCount=169, AvailableMemoryMB=6290 2023-04-19 20:53:40,596 DEBUG [Listener at localhost/34893] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-19 20:53:40,600 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:45258, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-19 20:53:40,609 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-19 20:53:40,614 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-19 20:53:40,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure2.ProcedureExecutor(1029): Stored pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-19 20:53:40,618 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-19 20:53:40,620 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-19 20:53:40,622 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 9 2023-04-19 20:53:40,623 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/.tmp/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5 2023-04-19 20:53:40,625 DEBUG [HFileArchiver-2] backup.HFileArchiver(153): Directory hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/.tmp/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5 empty. 2023-04-19 20:53:40,627 DEBUG [HFileArchiver-2] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/.tmp/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5 2023-04-19 20:53:40,627 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-19 20:53:40,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-19 20:53:40,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-19 20:53:40,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-19 20:53:41,045 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-19 20:53:41,047 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => f70bfe945d0054ce5a228cf255d902f5, NAME => 'TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/.tmp 2023-04-19 20:53:41,060 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-19 20:53:41,060 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing f70bfe945d0054ce5a228cf255d902f5, disabling compactions & flushes 2023-04-19 20:53:41,060 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. 2023-04-19 20:53:41,060 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. 2023-04-19 20:53:41,060 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. after waiting 0 ms 2023-04-19 20:53:41,060 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. 2023-04-19 20:53:41,060 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. 2023-04-19 20:53:41,060 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for f70bfe945d0054ce5a228cf255d902f5: 2023-04-19 20:53:41,065 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-19 20:53:41,067 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681937621066"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681937621066"}]},"ts":"1681937621066"} 2023-04-19 20:53:41,070 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-19 20:53:41,071 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-19 20:53:41,072 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681937621071"}]},"ts":"1681937621071"} 2023-04-19 20:53:41,074 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-19 20:53:41,078 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=f70bfe945d0054ce5a228cf255d902f5, ASSIGN}] 2023-04-19 20:53:41,080 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=f70bfe945d0054ce5a228cf255d902f5, ASSIGN 2023-04-19 20:53:41,081 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=f70bfe945d0054ce5a228cf255d902f5, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,35247,1681937616971; forceNewPlan=false, retain=false 2023-04-19 20:53:41,232 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=f70bfe945d0054ce5a228cf255d902f5, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:41,233 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681937621232"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681937621232"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681937621232"}]},"ts":"1681937621232"} 2023-04-19 20:53:41,237 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=11, ppid=10, state=RUNNABLE; OpenRegionProcedure f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971}] 2023-04-19 20:53:41,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-19 20:53:41,397 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. 2023-04-19 20:53:41,398 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => f70bfe945d0054ce5a228cf255d902f5, NAME => 'TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5.', STARTKEY => '', ENDKEY => ''} 2023-04-19 20:53:41,398 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees f70bfe945d0054ce5a228cf255d902f5 2023-04-19 20:53:41,398 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-19 20:53:41,398 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for f70bfe945d0054ce5a228cf255d902f5 2023-04-19 20:53:41,398 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for f70bfe945d0054ce5a228cf255d902f5 2023-04-19 20:53:41,400 INFO [StoreOpener-f70bfe945d0054ce5a228cf255d902f5-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region f70bfe945d0054ce5a228cf255d902f5 2023-04-19 20:53:41,402 DEBUG [StoreOpener-f70bfe945d0054ce5a228cf255d902f5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A 2023-04-19 20:53:41,403 DEBUG [StoreOpener-f70bfe945d0054ce5a228cf255d902f5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A 2023-04-19 20:53:41,404 INFO [StoreOpener-f70bfe945d0054ce5a228cf255d902f5-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-19 20:53:41,405 INFO [StoreOpener-f70bfe945d0054ce5a228cf255d902f5-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region f70bfe945d0054ce5a228cf255d902f5 columnFamilyName A 2023-04-19 20:53:41,405 INFO [StoreOpener-f70bfe945d0054ce5a228cf255d902f5-1] regionserver.HStore(310): Store=f70bfe945d0054ce5a228cf255d902f5/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-19 20:53:41,406 INFO [StoreOpener-f70bfe945d0054ce5a228cf255d902f5-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region f70bfe945d0054ce5a228cf255d902f5 2023-04-19 20:53:41,408 DEBUG [StoreOpener-f70bfe945d0054ce5a228cf255d902f5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B 2023-04-19 20:53:41,408 DEBUG [StoreOpener-f70bfe945d0054ce5a228cf255d902f5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B 2023-04-19 20:53:41,408 INFO [StoreOpener-f70bfe945d0054ce5a228cf255d902f5-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-19 20:53:41,408 INFO [StoreOpener-f70bfe945d0054ce5a228cf255d902f5-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region f70bfe945d0054ce5a228cf255d902f5 columnFamilyName B 2023-04-19 20:53:41,409 INFO [StoreOpener-f70bfe945d0054ce5a228cf255d902f5-1] regionserver.HStore(310): Store=f70bfe945d0054ce5a228cf255d902f5/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-19 20:53:41,409 INFO [StoreOpener-f70bfe945d0054ce5a228cf255d902f5-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region f70bfe945d0054ce5a228cf255d902f5 2023-04-19 20:53:41,411 DEBUG [StoreOpener-f70bfe945d0054ce5a228cf255d902f5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C 2023-04-19 20:53:41,411 DEBUG [StoreOpener-f70bfe945d0054ce5a228cf255d902f5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C 2023-04-19 20:53:41,411 INFO [StoreOpener-f70bfe945d0054ce5a228cf255d902f5-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-19 20:53:41,412 INFO [StoreOpener-f70bfe945d0054ce5a228cf255d902f5-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region f70bfe945d0054ce5a228cf255d902f5 columnFamilyName C 2023-04-19 20:53:41,412 INFO [StoreOpener-f70bfe945d0054ce5a228cf255d902f5-1] regionserver.HStore(310): Store=f70bfe945d0054ce5a228cf255d902f5/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-19 20:53:41,413 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. 2023-04-19 20:53:41,415 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5 2023-04-19 20:53:41,415 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5 2023-04-19 20:53:41,419 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-19 20:53:41,422 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for f70bfe945d0054ce5a228cf255d902f5 2023-04-19 20:53:41,425 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-19 20:53:41,426 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened f70bfe945d0054ce5a228cf255d902f5; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=66561770, jitterRate=-0.00815233588218689}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-19 20:53:41,426 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for f70bfe945d0054ce5a228cf255d902f5: 2023-04-19 20:53:41,427 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5., pid=11, masterSystemTime=1681937621391 2023-04-19 20:53:41,430 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. 2023-04-19 20:53:41,430 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. 2023-04-19 20:53:41,431 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=f70bfe945d0054ce5a228cf255d902f5, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:41,432 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681937621431"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681937621431"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681937621431"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681937621431"}]},"ts":"1681937621431"} 2023-04-19 20:53:41,439 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=11, resume processing ppid=10 2023-04-19 20:53:41,439 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=11, ppid=10, state=SUCCESS; OpenRegionProcedure f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 in 199 msec 2023-04-19 20:53:41,443 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=10, resume processing ppid=9 2023-04-19 20:53:41,443 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=10, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=f70bfe945d0054ce5a228cf255d902f5, ASSIGN in 361 msec 2023-04-19 20:53:41,444 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-19 20:53:41,445 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681937621444"}]},"ts":"1681937621444"} 2023-04-19 20:53:41,447 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-19 20:53:41,451 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-19 20:53:41,453 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=9, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 837 msec 2023-04-19 20:53:41,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-19 20:53:41,745 INFO [Listener at localhost/34893] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 9 completed 2023-04-19 20:53:41,750 DEBUG [Listener at localhost/34893] zookeeper.ReadOnlyZKClient(139): Connect 0x2e0a05b0 to 127.0.0.1:58736 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-19 20:53:41,763 DEBUG [Listener at localhost/34893] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@48da4d35, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-19 20:53:41,767 DEBUG [Listener at localhost/34893] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-19 20:53:41,772 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:42312, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-19 20:53:41,775 DEBUG [Listener at localhost/34893] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-19 20:53:41,782 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:45266, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-19 20:53:41,791 DEBUG [Listener at localhost/34893] zookeeper.ReadOnlyZKClient(139): Connect 0x402c2635 to 127.0.0.1:58736 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-19 20:53:41,803 DEBUG [Listener at localhost/34893] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@72ea4a65, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-19 20:53:41,804 DEBUG [Listener at localhost/34893] zookeeper.ReadOnlyZKClient(139): Connect 0x377933e4 to 127.0.0.1:58736 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-19 20:53:41,810 DEBUG [Listener at localhost/34893] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@36dc7f7f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-19 20:53:41,810 DEBUG [Listener at localhost/34893] zookeeper.ReadOnlyZKClient(139): Connect 0x401974c7 to 127.0.0.1:58736 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-19 20:53:41,815 DEBUG [Listener at localhost/34893] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7b52aa42, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-19 20:53:41,815 DEBUG [Listener at localhost/34893] zookeeper.ReadOnlyZKClient(139): Connect 0x1ca4a062 to 127.0.0.1:58736 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-19 20:53:41,822 DEBUG [Listener at localhost/34893] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@79fee867, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-19 20:53:41,822 DEBUG [Listener at localhost/34893] zookeeper.ReadOnlyZKClient(139): Connect 0x6ec07124 to 127.0.0.1:58736 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-19 20:53:41,833 DEBUG [Listener at localhost/34893] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7bc6d729, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-19 20:53:41,834 DEBUG [Listener at localhost/34893] zookeeper.ReadOnlyZKClient(139): Connect 0x29fba56a to 127.0.0.1:58736 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-19 20:53:41,842 DEBUG [Listener at localhost/34893] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@375c9d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-19 20:53:41,843 DEBUG [Listener at localhost/34893] zookeeper.ReadOnlyZKClient(139): Connect 0x5ae6c42b to 127.0.0.1:58736 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-19 20:53:41,872 DEBUG [Listener at localhost/34893] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2ec2fae8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-19 20:53:41,872 DEBUG [Listener at localhost/34893] zookeeper.ReadOnlyZKClient(139): Connect 0x1e44f1b3 to 127.0.0.1:58736 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-19 20:53:41,878 DEBUG [Listener at localhost/34893] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@27685d29, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-19 20:53:41,878 DEBUG [Listener at localhost/34893] zookeeper.ReadOnlyZKClient(139): Connect 0x6089dec1 to 127.0.0.1:58736 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-19 20:53:41,882 DEBUG [Listener at localhost/34893] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5979620a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-19 20:53:41,895 DEBUG [hconnection-0x1531940c-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-19 20:53:41,898 DEBUG [hconnection-0x6d1c3ce9-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-19 20:53:41,899 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:42316, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-19 20:53:41,899 DEBUG [hconnection-0x637b85c0-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-19 20:53:41,899 DEBUG [hconnection-0x628adf72-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-19 20:53:41,900 DEBUG [hconnection-0x26902f80-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-19 20:53:41,905 DEBUG [hconnection-0xff560f1-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-19 20:53:41,906 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:42330, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-19 20:53:41,905 DEBUG [hconnection-0x66595227-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-19 20:53:41,907 DEBUG [hconnection-0x66fde656-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-19 20:53:41,907 DEBUG [hconnection-0x1516ed88-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-19 20:53:41,910 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:42350, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-19 20:53:41,912 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:42342, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-19 20:53:41,912 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:42344, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-19 20:53:41,913 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:53:41,916 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:42358, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-19 20:53:41,917 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:42380, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-19 20:53:41,919 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:42366, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-19 20:53:41,926 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:42384, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-19 20:53:41,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:53:41,933 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:53:41,933 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:53:41,941 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:53:41,941 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:53:41,942 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:41,943 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:41,945 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:53:41,945 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:41,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:53:41,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:53:41,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:41,946 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:41,946 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:53:41,947 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:41,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:53:41,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:41,948 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:53:41,952 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:53:41,956 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:53:41,956 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:53:41,957 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:53:41,959 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:53:41,959 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:53:41,959 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. 2023-04-19 20:53:41,961 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. started... 2023-04-19 20:53:41,986 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing f70bfe945d0054ce5a228cf255d902f5 3/3 column families, dataSize=20.13 KB heapSize=60.52 KB 2023-04-19 20:53:41,998 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f70bfe945d0054ce5a228cf255d902f5, store=A 2023-04-19 20:53:41,999 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:53:42,001 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f70bfe945d0054ce5a228cf255d902f5, store=B 2023-04-19 20:53:42,002 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:53:42,002 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f70bfe945d0054ce5a228cf255d902f5, store=C 2023-04-19 20:53:42,002 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:53:42,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(9158): Flush requested on f70bfe945d0054ce5a228cf255d902f5 2023-04-19 20:53:42,028 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. as already flushing 2023-04-19 20:53:42,146 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681937622066, max timestamp=1681937622116Num uniques 859; 2023-04-19 20:53:42,147 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 20:53:42,147 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681937622066, max timestamp=1681937622122Num uniques 859; 2023-04-19 20:53:42,148 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 20:53:42,162 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681937622066, max timestamp=1681937622116Num uniques 859; 2023-04-19 20:53:42,162 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-19 20:53:42,276 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7680, totalHeapSize=17712, min timestamp=1681937622246, max timestamp=1681937622266Num uniques 1018; 2023-04-19 20:53:42,276 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7680, totalHeapSize=17712, min timestamp=1681937622246, max timestamp=1681937622266Num uniques 1018; 2023-04-19 20:53:42,276 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46610, new segment data size=7680, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-19 20:53:42,277 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681937622246, max timestamp=1681937622264Num uniques 1009; 2023-04-19 20:53:42,277 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46213, new segment data size=7688, suffix heap size=132904,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=168 2023-04-19 20:53:42,276 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46610, new segment data size=7680, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-19 20:53:42,337 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681937622300, max timestamp=1681937622319Num uniques 1027; 2023-04-19 20:53:42,338 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=6870, suffix heap size=134776,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=150 2023-04-19 20:53:42,338 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681937622300, max timestamp=1681937622312Num uniques 1027; 2023-04-19 20:53:42,338 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1681937622300, max timestamp=1681937622315Num uniques 1027; 2023-04-19 20:53:42,338 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-19 20:53:42,338 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=6870, suffix heap size=134776,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=150 2023-04-19 20:53:42,376 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1681937622354, max timestamp=1681937622370Num uniques 1009; 2023-04-19 20:53:42,376 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=8516, suffix heap size=132904,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=186 2023-04-19 20:53:42,377 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1681937622354, max timestamp=1681937622370Num uniques 1036; 2023-04-19 20:53:42,377 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-04-19 20:53:42,378 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1681937622354, max timestamp=1681937622370Num uniques 1009; 2023-04-19 20:53:42,378 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=8516, suffix heap size=132904,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=186 2023-04-19 20:53:42,422 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1681937622407, max timestamp=1681937622417Num uniques 1045; 2023-04-19 20:53:42,422 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1681937622407, max timestamp=1681937622417Num uniques 1045; 2023-04-19 20:53:42,422 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=11220, suffix heap size=136648,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=245 2023-04-19 20:53:42,422 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1681937622407, max timestamp=1681937622417Num uniques 1045; 2023-04-19 20:53:42,422 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=11220, suffix heap size=136648,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=245 2023-04-19 20:53:42,422 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=11220, suffix heap size=136648,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=245 2023-04-19 20:53:42,467 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=304, cellSize=13914, totalHeapSize=31856, min timestamp=1681937622407, max timestamp=1681937622463Num uniques 1104; 2023-04-19 20:53:42,468 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=50554, new segment data size=13914, suffix heap size=142784,new segment heap size=31856 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=304 2023-04-19 20:53:42,468 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=304, cellSize=13915, totalHeapSize=31856, min timestamp=1681937622407, max timestamp=1681937622463Num uniques 1104; 2023-04-19 20:53:42,468 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=304, cellSize=13917, totalHeapSize=31856, min timestamp=1681937622407, max timestamp=1681937622463Num uniques 1104; 2023-04-19 20:53:42,468 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=50555, new segment data size=13915, suffix heap size=142784,new segment heap size=31856 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=304 2023-04-19 20:53:42,468 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=50557, new segment data size=13917, suffix heap size=142784,new segment heap size=31856 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=304 2023-04-19 20:53:42,516 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:42,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42350 deadline: 1681937682508, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:42,518 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:42,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42366 deadline: 1681937682511, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:42,518 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1681937622495, max timestamp=1681937622511Num uniques 1163; 2023-04-19 20:53:42,518 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1681937622495, max timestamp=1681937622511Num uniques 1163; 2023-04-19 20:53:42,518 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=53258, new segment data size=7458, suffix heap size=148920,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1163, new segment cells count=163 2023-04-19 20:53:42,518 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=53258, new segment data size=7458, suffix heap size=148920,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1163, new segment cells count=163 2023-04-19 20:53:42,519 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1681937622495, max timestamp=1681937622510Num uniques 1163; 2023-04-19 20:53:42,519 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=53258, new segment data size=7458, suffix heap size=148920,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1163, new segment cells count=163 2023-04-19 20:53:42,529 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=7 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/A/c5820d14b7d445a39e49141037c74f93 2023-04-19 20:53:42,575 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=272, cellSize=12452, totalHeapSize=28528, min timestamp=1681937622548, max timestamp=1681937622573Num uniques 1022; 2023-04-19 20:53:42,576 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=12452, suffix heap size=134256,new segment heap size=28528 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=272 2023-04-19 20:53:42,576 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=272, cellSize=12452, totalHeapSize=28528, min timestamp=1681937622548, max timestamp=1681937622573Num uniques 1022; 2023-04-19 20:53:42,577 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=12452, suffix heap size=134256,new segment heap size=28528 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=272 2023-04-19 20:53:42,579 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=272, cellSize=12452, totalHeapSize=28528, min timestamp=1681937622548, max timestamp=1681937622573Num uniques 1022; 2023-04-19 20:53:42,580 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=12452, suffix heap size=134256,new segment heap size=28528 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=272 2023-04-19 20:53:42,626 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=431, cellSize=19736, totalHeapSize=45064, min timestamp=1681937622600, max timestamp=1681937622622Num uniques 1131; 2023-04-19 20:53:42,627 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=51796, new segment data size=19736, suffix heap size=145592,new segment heap size=45064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1131, new segment cells count=431 2023-04-19 20:53:42,631 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681937622615, max timestamp=1681937622622Num uniques 1131; 2023-04-19 20:53:42,631 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=51796, new segment data size=6870, suffix heap size=145592,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1131, new segment cells count=150 2023-04-19 20:53:42,634 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=431, cellSize=19736, totalHeapSize=45064, min timestamp=1681937622600, max timestamp=1681937622622Num uniques 1131; 2023-04-19 20:53:42,634 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=51796, new segment data size=19736, suffix heap size=145592,new segment heap size=45064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1131, new segment cells count=431 2023-04-19 20:53:42,676 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=190, cellSize=8690, totalHeapSize=20000, min timestamp=1681937622657, max timestamp=1681937622672Num uniques 1290; 2023-04-19 20:53:42,677 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=190, cellSize=8700, totalHeapSize=20000, min timestamp=1681937622657, max timestamp=1681937622672Num uniques 1009; 2023-04-19 20:53:42,677 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=59070, new segment data size=8690, suffix heap size=162128,new segment heap size=20000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1290, new segment cells count=190 2023-04-19 20:53:42,677 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=190, cellSize=8691, totalHeapSize=20000, min timestamp=1681937622657, max timestamp=1681937622672Num uniques 1290; 2023-04-19 20:53:42,677 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=59071, new segment data size=8691, suffix heap size=162128,new segment heap size=20000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1290, new segment cells count=190 2023-04-19 20:53:42,677 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=8700, suffix heap size=132904,new segment heap size=20000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=190 2023-04-19 20:53:42,704 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=299, cellSize=13694, totalHeapSize=31336, min timestamp=1681937622678, max timestamp=1681937622701Num uniques 1049; 2023-04-19 20:53:42,704 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=48044, new segment data size=13694, suffix heap size=137064,new segment heap size=31336 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1049, new segment cells count=299 2023-04-19 20:53:42,704 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=299, cellSize=13694, totalHeapSize=31336, min timestamp=1681937622678, max timestamp=1681937622701Num uniques 1049; 2023-04-19 20:53:42,704 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=48044, new segment data size=13694, suffix heap size=137064,new segment heap size=31336 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1049, new segment cells count=299 2023-04-19 20:53:42,705 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9114, totalHeapSize=20936, min timestamp=1681937622690, max timestamp=1681937622701Num uniques 1049; 2023-04-19 20:53:42,705 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=48044, new segment data size=9114, suffix heap size=137064,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1049, new segment cells count=199 2023-04-19 20:53:42,735 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:42,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42384 deadline: 1681937682729, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:42,735 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=158, cellSize=7230, totalHeapSize=16672, min timestamp=1681937622720, max timestamp=1681937622732Num uniques 1158; 2023-04-19 20:53:42,735 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:42,736 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=158, cellSize=7230, totalHeapSize=16672, min timestamp=1681937622720, max timestamp=1681937622732Num uniques 1158; 2023-04-19 20:53:42,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42350 deadline: 1681937682733, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:42,736 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=53030, new segment data size=7230, suffix heap size=148400,new segment heap size=16672 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1158, new segment cells count=158 2023-04-19 20:53:42,735 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=53030, new segment data size=7230, suffix heap size=148400,new segment heap size=16672 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1158, new segment cells count=158 2023-04-19 20:53:42,737 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681937622720, max timestamp=1681937622732Num uniques 1058; 2023-04-19 20:53:42,737 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=48450, new segment data size=6870, suffix heap size=138000,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1058, new segment cells count=150 2023-04-19 20:53:42,757 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=167, cellSize=7642, totalHeapSize=17608, min timestamp=1681937622746, max timestamp=1681937622754Num uniques 1017; 2023-04-19 20:53:42,758 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46572, new segment data size=7642, suffix heap size=133736,new segment heap size=17608 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1017, new segment cells count=167 2023-04-19 20:53:42,758 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=167, cellSize=7642, totalHeapSize=17608, min timestamp=1681937622746, max timestamp=1681937622754Num uniques 1017; 2023-04-19 20:53:42,758 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46572, new segment data size=7642, suffix heap size=133736,new segment heap size=17608 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1017, new segment cells count=167 2023-04-19 20:53:42,760 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=167, cellSize=7639, totalHeapSize=17608, min timestamp=1681937622746, max timestamp=1681937622755Num uniques 1009; 2023-04-19 20:53:42,761 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46209, new segment data size=7639, suffix heap size=132904,new segment heap size=17608 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=167 2023-04-19 20:53:42,790 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=226, cellSize=10346, totalHeapSize=23744, min timestamp=1681937622774, max timestamp=1681937622786Num uniques 1026; 2023-04-19 20:53:42,790 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46986, new segment data size=10346, suffix heap size=134672,new segment heap size=23744 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1026, new segment cells count=226 2023-04-19 20:53:42,792 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=176, cellSize=8056, totalHeapSize=18544, min timestamp=1681937622783, max timestamp=1681937622786Num uniques 1026; 2023-04-19 20:53:42,792 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46986, new segment data size=8056, suffix heap size=134672,new segment heap size=18544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1026, new segment cells count=176 2023-04-19 20:53:42,795 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=176, cellSize=8056, totalHeapSize=18544, min timestamp=1681937622783, max timestamp=1681937622786Num uniques 1026; 2023-04-19 20:53:42,795 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46986, new segment data size=8056, suffix heap size=134672,new segment heap size=18544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1026, new segment cells count=176 2023-04-19 20:53:42,835 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=285, cellSize=13050, totalHeapSize=29880, min timestamp=1681937622808, max timestamp=1681937622832Num uniques 1085; 2023-04-19 20:53:42,835 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=49690, new segment data size=13050, suffix heap size=140808,new segment heap size=29880 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1085, new segment cells count=285 2023-04-19 20:53:42,835 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=285, cellSize=13050, totalHeapSize=29880, min timestamp=1681937622808, max timestamp=1681937622832Num uniques 1035; 2023-04-19 20:53:42,835 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47400, new segment data size=13050, suffix heap size=135608,new segment heap size=29880 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1035, new segment cells count=285 2023-04-19 20:53:42,837 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=285, cellSize=13050, totalHeapSize=29880, min timestamp=1681937622808, max timestamp=1681937622832Num uniques 1035; 2023-04-19 20:53:42,837 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47400, new segment data size=13050, suffix heap size=135608,new segment heap size=29880 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1035, new segment cells count=285 2023-04-19 20:53:42,893 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=244, cellSize=11174, totalHeapSize=25616, min timestamp=1681937622857, max timestamp=1681937622890Num uniques 1144; 2023-04-19 20:53:42,893 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=244, cellSize=11174, totalHeapSize=25616, min timestamp=1681937622857, max timestamp=1681937622890Num uniques 1144; 2023-04-19 20:53:42,893 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=244, cellSize=11174, totalHeapSize=25616, min timestamp=1681937622857, max timestamp=1681937622890Num uniques 1144; 2023-04-19 20:53:42,893 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=52394, new segment data size=11174, suffix heap size=146944,new segment heap size=25616 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1144, new segment cells count=244 2023-04-19 20:53:42,893 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=52394, new segment data size=11174, suffix heap size=146944,new segment heap size=25616 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1144, new segment cells count=244 2023-04-19 20:53:42,893 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=52394, new segment data size=11174, suffix heap size=146944,new segment heap size=25616 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1144, new segment cells count=244 2023-04-19 20:53:42,921 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=303, cellSize=13875, totalHeapSize=31752, min timestamp=1681937622902, max timestamp=1681937622917Num uniques 1103; 2023-04-19 20:53:42,921 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=303, cellSize=13875, totalHeapSize=31752, min timestamp=1681937622902, max timestamp=1681937622917Num uniques 1103; 2023-04-19 20:53:42,921 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=50515, new segment data size=13875, suffix heap size=142680,new segment heap size=31752 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1103, new segment cells count=303 2023-04-19 20:53:42,923 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=303, cellSize=13875, totalHeapSize=31752, min timestamp=1681937622902, max timestamp=1681937622917Num uniques 1103; 2023-04-19 20:53:42,923 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=50515, new segment data size=13875, suffix heap size=142680,new segment heap size=31752 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1103, new segment cells count=303 2023-04-19 20:53:42,921 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=50515, new segment data size=13875, suffix heap size=142680,new segment heap size=31752 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1103, new segment cells count=303 2023-04-19 20:53:42,946 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=212, cellSize=9702, totalHeapSize=22288, min timestamp=1681937622933, max timestamp=1681937622943Num uniques 1162; 2023-04-19 20:53:42,946 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=212, cellSize=9702, totalHeapSize=22288, min timestamp=1681937622933, max timestamp=1681937622943Num uniques 1162; 2023-04-19 20:53:42,946 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=53212, new segment data size=9702, suffix heap size=148816,new segment heap size=22288 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1162, new segment cells count=212 2023-04-19 20:53:42,946 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=53212, new segment data size=9702, suffix heap size=148816,new segment heap size=22288 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1162, new segment cells count=212 2023-04-19 20:53:42,948 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=162, cellSize=7412, totalHeapSize=17088, min timestamp=1681937622939, max timestamp=1681937622943Num uniques 1162; 2023-04-19 20:53:42,949 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=53212, new segment data size=7412, suffix heap size=148816,new segment heap size=17088 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1162, new segment cells count=162 2023-04-19 20:53:42,978 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=171, cellSize=7826, totalHeapSize=18024, min timestamp=1681937622967, max timestamp=1681937622972Num uniques 1071; 2023-04-19 20:53:42,978 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681937622967, max timestamp=1681937622972Num uniques 1021; 2023-04-19 20:53:42,979 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46756, new segment data size=6870, suffix heap size=134152,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1021, new segment cells count=150 2023-04-19 20:53:42,978 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=49046, new segment data size=7826, suffix heap size=139352,new segment heap size=18024 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1071, new segment cells count=171 2023-04-19 20:53:42,980 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=171, cellSize=7826, totalHeapSize=18024, min timestamp=1681937622967, max timestamp=1681937622972Num uniques 1071; 2023-04-19 20:53:42,980 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=49046, new segment data size=7826, suffix heap size=139352,new segment heap size=18024 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1071, new segment cells count=171 2023-04-19 20:53:43,005 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=180, cellSize=8240, totalHeapSize=18960, min timestamp=1681937622992, max timestamp=1681937623001Num uniques 1030; 2023-04-19 20:53:43,005 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47170, new segment data size=8240, suffix heap size=135088,new segment heap size=18960 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1030, new segment cells count=180 2023-04-19 20:53:43,005 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=180, cellSize=8240, totalHeapSize=18960, min timestamp=1681937622992, max timestamp=1681937623001Num uniques 1030; 2023-04-19 20:53:43,005 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=47170, new segment data size=8240, suffix heap size=135088,new segment heap size=18960 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1030, new segment cells count=180 2023-04-19 20:53:43,005 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=180, cellSize=8240, totalHeapSize=18960, min timestamp=1681937622992, max timestamp=1681937623002Num uniques 1009; 2023-04-19 20:53:43,006 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=8240, suffix heap size=132904,new segment heap size=18960 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=180 2023-04-19 20:53:43,026 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=239, cellSize=10944, totalHeapSize=25096, min timestamp=1681937623021, max timestamp=1681937623023Num uniques 1039; 2023-04-19 20:53:43,027 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=239, cellSize=10944, totalHeapSize=25096, min timestamp=1681937623021, max timestamp=1681937623023Num uniques 1039; 2023-04-19 20:53:43,027 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=47584, new segment data size=10944, suffix heap size=136024,new segment heap size=25096 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1039, new segment cells count=239 2023-04-19 20:53:43,027 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=239, cellSize=10944, totalHeapSize=25096, min timestamp=1681937623021, max timestamp=1681937623023Num uniques 1039; 2023-04-19 20:53:43,027 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47584, new segment data size=10944, suffix heap size=136024,new segment heap size=25096 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1039, new segment cells count=239 2023-04-19 20:53:43,027 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=47584, new segment data size=10944, suffix heap size=136024,new segment heap size=25096 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1039, new segment cells count=239 2023-04-19 20:53:43,056 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=248, cellSize=11358, totalHeapSize=26032, min timestamp=1681937623048, max timestamp=1681937623054Num uniques 1098; 2023-04-19 20:53:43,056 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=50288, new segment data size=11358, suffix heap size=142160,new segment heap size=26032 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1098, new segment cells count=248 2023-04-19 20:53:43,057 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=248, cellSize=11358, totalHeapSize=26032, min timestamp=1681937623048, max timestamp=1681937623054Num uniques 1098; 2023-04-19 20:53:43,058 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=50288, new segment data size=11358, suffix heap size=142160,new segment heap size=26032 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1098, new segment cells count=248 2023-04-19 20:53:43,066 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=248, cellSize=11358, totalHeapSize=26032, min timestamp=1681937623048, max timestamp=1681937623055Num uniques 1098; 2023-04-19 20:53:43,067 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=50288, new segment data size=11358, suffix heap size=142160,new segment heap size=26032 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1098, new segment cells count=248 2023-04-19 20:53:43,072 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=7 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/B/daf9346af1d1405c96ef9a592f36872e 2023-04-19 20:53:43,107 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=207, cellSize=9475, totalHeapSize=21768, min timestamp=1681937623093, max timestamp=1681937623104Num uniques 1107; 2023-04-19 20:53:43,107 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=207, cellSize=9475, totalHeapSize=21768, min timestamp=1681937623093, max timestamp=1681937623104Num uniques 1107; 2023-04-19 20:53:43,107 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=50695, new segment data size=9475, suffix heap size=143096,new segment heap size=21768 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1107, new segment cells count=207 2023-04-19 20:53:43,107 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=207, cellSize=9475, totalHeapSize=21768, min timestamp=1681937623093, max timestamp=1681937623104Num uniques 1107; 2023-04-19 20:53:43,107 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=50695, new segment data size=9475, suffix heap size=143096,new segment heap size=21768 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1107, new segment cells count=207 2023-04-19 20:53:43,107 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=50695, new segment data size=9475, suffix heap size=143096,new segment heap size=21768 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1107, new segment cells count=207 2023-04-19 20:53:43,139 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=266, cellSize=12176, totalHeapSize=27904, min timestamp=1681937623117, max timestamp=1681937623135Num uniques 1066; 2023-04-19 20:53:43,139 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=266, cellSize=12176, totalHeapSize=27904, min timestamp=1681937623117, max timestamp=1681937623135Num uniques 1066; 2023-04-19 20:53:43,139 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=48816, new segment data size=12176, suffix heap size=138832,new segment heap size=27904 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1066, new segment cells count=266 2023-04-19 20:53:43,139 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=266, cellSize=12176, totalHeapSize=27904, min timestamp=1681937623117, max timestamp=1681937623135Num uniques 1066; 2023-04-19 20:53:43,140 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=48816, new segment data size=12176, suffix heap size=138832,new segment heap size=27904 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1066, new segment cells count=266 2023-04-19 20:53:43,139 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=48816, new segment data size=12176, suffix heap size=138832,new segment heap size=27904 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1066, new segment cells count=266 2023-04-19 20:53:43,159 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=7 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/C/0fb95bdc773047e3b2042cbe5f94f3c0 2023-04-19 20:53:43,172 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/A/c5820d14b7d445a39e49141037c74f93 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/c5820d14b7d445a39e49141037c74f93 2023-04-19 20:53:43,187 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=225, cellSize=10300, totalHeapSize=23640, min timestamp=1681937623169, max timestamp=1681937623182Num uniques 1125; 2023-04-19 20:53:43,187 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=51520, new segment data size=10300, suffix heap size=144968,new segment heap size=23640 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1125, new segment cells count=225 2023-04-19 20:53:43,187 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=225, cellSize=10300, totalHeapSize=23640, min timestamp=1681937623169, max timestamp=1681937623184Num uniques 1125; 2023-04-19 20:53:43,189 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=51520, new segment data size=10300, suffix heap size=144968,new segment heap size=23640 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1125, new segment cells count=225 2023-04-19 20:53:43,190 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/c5820d14b7d445a39e49141037c74f93, entries=100, sequenceid=7, filesize=9.3 K 2023-04-19 20:53:43,190 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=225, cellSize=10300, totalHeapSize=23640, min timestamp=1681937623169, max timestamp=1681937623184Num uniques 1125; 2023-04-19 20:53:43,191 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=51520, new segment data size=10300, suffix heap size=144968,new segment heap size=23640 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1125, new segment cells count=225 2023-04-19 20:53:43,195 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/B/daf9346af1d1405c96ef9a592f36872e as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/daf9346af1d1405c96ef9a592f36872e 2023-04-19 20:53:43,222 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/daf9346af1d1405c96ef9a592f36872e, entries=100, sequenceid=7, filesize=9.3 K 2023-04-19 20:53:43,226 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/C/0fb95bdc773047e3b2042cbe5f94f3c0 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/0fb95bdc773047e3b2042cbe5f94f3c0 2023-04-19 20:53:43,257 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/0fb95bdc773047e3b2042cbe5f94f3c0, entries=100, sequenceid=7, filesize=9.3 K 2023-04-19 20:53:43,269 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.13 KB/20610, heapSize ~60.47 KB/61920, currentSize=87.22 KB/89310 for f70bfe945d0054ce5a228cf255d902f5 in 1306ms, sequenceid=7, compaction requested=false 2023-04-19 20:53:43,271 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-04-19 20:53:43,274 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f70bfe945d0054ce5a228cf255d902f5: 2023-04-19 20:53:43,274 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. 2023-04-19 20:53:43,274 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:53:43,275 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:53:43,276 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:43,276 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:53:43,276 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:53:43,280 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:43,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:43,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:53:43,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:53:43,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:53:43,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(9158): Flush requested on f70bfe945d0054ce5a228cf255d902f5 2023-04-19 20:53:43,286 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing f70bfe945d0054ce5a228cf255d902f5 3/3 column families, dataSize=93.93 KB heapSize=259.27 KB 2023-04-19 20:53:43,286 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f70bfe945d0054ce5a228cf255d902f5, store=A 2023-04-19 20:53:43,287 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 20:53:43,287 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f70bfe945d0054ce5a228cf255d902f5, store=B 2023-04-19 20:53:43,287 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 20:53:43,287 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f70bfe945d0054ce5a228cf255d902f5, store=C 2023-04-19 20:53:43,287 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 20:53:43,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:53:43,288 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:43,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:43,289 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:43,289 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:53:43,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:43,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:53:43,291 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:53:43,291 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7e630da0[Count = 0] remaining members to acquire global barrier 2023-04-19 20:53:43,291 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:53:43,291 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:43,295 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:43,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:43,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:43,300 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:53:43,300 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:53:43,300 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:53:43,307 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:53:43,307 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:53:43,307 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:53:43,309 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:43,310 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:53:43,310 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:53:43,310 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:53:43,310 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:53:43,310 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:53:43,310 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:53:43,310 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:53:43,325 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:43,325 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:43,325 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:53:43,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:43,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:53:43,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:53:43,326 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:43,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:43,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:53:43,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:53:43,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:53:43,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:43,328 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:43,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:43,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:53:43,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:43,331 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:43,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:43,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:53:43,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:43,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:43,366 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:53:43,366 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:53:43,366 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:43,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:53:43,366 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:43,366 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:53:43,367 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:43,366 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:53:43,370 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:53:43,370 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=449 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/A/6d7e233041aa47a8a61054b74f3c78e3 2023-04-19 20:53:43,370 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:43,372 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:43,372 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:43,372 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:43,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:53:43,372 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:53:43,372 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:53:43,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:53:43,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:53:43,378 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:43,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42366 deadline: 1681937683328, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:43,380 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:43,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42380 deadline: 1681937683331, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:43,403 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:43,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42384 deadline: 1681937683376, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:43,404 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:43,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42342 deadline: 1681937683377, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:43,404 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:43,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42350 deadline: 1681937683378, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:43,443 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=449 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/B/f02035798baf4d76a1a8ef590e665f2e 2023-04-19 20:53:43,473 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:53:43,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:53:43,484 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:43,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42366 deadline: 1681937683482, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:43,485 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:43,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42380 deadline: 1681937683482, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:43,487 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:53:43,511 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:43,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42350 deadline: 1681937683507, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:43,512 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:43,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42342 deadline: 1681937683508, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:43,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:53:43,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:53:43,516 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:53:43,516 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:53:43,522 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:53:43,523 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:53:43,524 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:43,524 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:43,527 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:53:43,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:53:43,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:53:43,528 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:43,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:43,529 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:43,529 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:43,529 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:53:43,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:53:43,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:43,530 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:53:43,535 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:43,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42384 deadline: 1681937683519, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:43,538 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:53:43,541 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:53:43,541 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:53:43,541 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:53:43,551 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:53:43,551 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:53:43,551 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. 2023-04-19 20:53:43,551 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:53:43,552 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. started... 2023-04-19 20:53:43,552 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. as already flushing 2023-04-19 20:53:43,552 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. 2023-04-19 20:53:43,557 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:53:43,557 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:53:43,557 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:43,558 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:53:43,558 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:53:43,561 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:43,561 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:43,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:43,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:53:43,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:53:43,562 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:43,562 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:53:43,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:53:43,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:53:43,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:43,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:43,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:53:43,565 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:53:43,565 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@ebb6d0f[Count = 0] remaining members to acquire global barrier 2023-04-19 20:53:43,565 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:53:43,565 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:43,566 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:43,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:43,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:43,567 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:53:43,567 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:53:43,567 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:53:43,567 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:43,567 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:53:43,570 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:53:43,570 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:53:43,570 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:53:43,570 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:43,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:43,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:53:43,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:53:43,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:53:43,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:53:43,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:43,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:43,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:53:43,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:43,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:43,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:53:43,576 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:53:43,576 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:53:43,576 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:53:43,576 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:53:43,576 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:53:43,576 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:53:43,582 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:43,582 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:43,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:43,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:43,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:53:43,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:53:43,582 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:53:43,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:43,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:53:43,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:53:43,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:53:43,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:43,584 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:43,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:43,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:53:43,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:43,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:43,587 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:43,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:53:43,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:43,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:43,596 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:43,596 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:43,596 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:53:43,596 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:53:43,596 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:43,596 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:53:43,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:53:43,596 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:53:43,596 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:53:43,601 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:53:43,596 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:53:43,596 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:43,601 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:43,601 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:43,601 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:43,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:53:43,602 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:53:43,602 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:53:43,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:53:43,698 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:43,700 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:43,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42366 deadline: 1681937683690, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:43,702 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:53:43,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:53:43,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42380 deadline: 1681937683689, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:43,707 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:53:43,719 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:43,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42342 deadline: 1681937683717, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:43,720 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:43,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42350 deadline: 1681937683718, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:43,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:53:43,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:53:43,728 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:53:43,728 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:53:43,733 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:53:43,733 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:53:43,734 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:43,734 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:43,737 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:53:43,737 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:43,737 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:53:43,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:53:43,738 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:43,738 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:53:43,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:43,739 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:43,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:53:43,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:43,740 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:53:43,742 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:53:43,742 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:53:43,742 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:53:43,743 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:53:43,745 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:53:43,745 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:53:43,745 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:53:43,746 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:43,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42384 deadline: 1681937683741, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:43,748 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. 2023-04-19 20:53:43,748 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. started... 2023-04-19 20:53:43,749 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. as already flushing 2023-04-19 20:53:43,749 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. 2023-04-19 20:53:43,750 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:53:43,750 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:53:43,750 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:43,750 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:53:43,750 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:53:43,753 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:43,753 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:43,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:43,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:53:43,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:53:43,754 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:43,754 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:53:43,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:53:43,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:53:43,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:43,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:43,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:53:43,759 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:53:43,760 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@772cdbe8[Count = 0] remaining members to acquire global barrier 2023-04-19 20:53:43,760 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:53:43,760 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:43,762 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:43,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:43,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:43,763 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:43,763 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:53:43,763 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:53:43,763 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:53:43,763 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:53:43,769 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:43,769 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:53:43,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:43,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:53:43,769 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:53:43,769 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:53:43,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:53:43,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:53:43,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:53:43,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:43,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:43,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:53:43,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:43,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:43,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:53:43,777 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:53:43,777 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:53:43,777 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:53:43,777 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:53:43,777 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:53:43,777 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:53:43,780 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:43,781 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:53:43,781 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:43,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:43,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:43,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:53:43,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:53:43,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:43,782 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:43,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:53:43,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:53:43,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:53:43,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:43,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:43,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:53:43,785 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:43,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:43,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:43,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:53:43,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:43,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:43,799 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:53:43,799 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:53:43,799 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:53:43,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:53:43,799 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:53:43,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:53:43,804 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:43,804 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:43,804 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:43,804 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:53:43,805 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:53:43,804 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:43,804 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:53:43,805 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:43,805 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:43,805 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:43,805 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:53:43,805 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:53:43,805 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:53:43,906 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:53:43,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:53:43,912 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:53:43,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:53:43,952 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=449 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/C/14d26d1a0f2749e18f2758e1568b64fb 2023-04-19 20:53:43,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:53:43,953 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:53:43,953 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:53:43,961 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:53:43,961 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:53:43,962 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:43,962 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:43,965 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:53:43,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:53:43,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:53:43,965 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:43,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:43,968 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:43,968 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:53:43,968 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:43,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:53:43,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:43,971 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:53:43,972 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:53:43,972 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:53:43,972 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:53:43,972 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:53:43,983 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:53:43,985 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/A/6d7e233041aa47a8a61054b74f3c78e3 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/6d7e233041aa47a8a61054b74f3c78e3 2023-04-19 20:53:43,988 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:53:43,988 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:53:43,992 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. 2023-04-19 20:53:43,992 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. started... 2023-04-19 20:53:43,992 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. as already flushing 2023-04-19 20:53:43,992 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. 2023-04-19 20:53:43,994 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:53:43,994 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:53:43,994 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:43,994 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:53:43,994 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:53:44,004 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:44,004 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:44,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:44,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:53:44,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:53:44,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:53:44,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:53:44,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:44,007 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:44,007 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:53:44,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:44,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:53:44,009 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:53:44,009 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@39a80319[Count = 0] remaining members to acquire global barrier 2023-04-19 20:53:44,009 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:53:44,009 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:44,010 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/6d7e233041aa47a8a61054b74f3c78e3, entries=450, sequenceid=449, filesize=26.2 K 2023-04-19 20:53:44,011 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:44,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:44,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:44,011 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:53:44,011 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:44,011 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:53:44,011 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:53:44,011 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:53:44,017 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/B/f02035798baf4d76a1a8ef590e665f2e as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/f02035798baf4d76a1a8ef590e665f2e 2023-04-19 20:53:44,025 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:44,026 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:53:44,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:44,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:53:44,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:53:44,026 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:53:44,026 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:53:44,028 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:44,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42366 deadline: 1681937684006, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:44,037 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:44,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42380 deadline: 1681937684011, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:44,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:53:44,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:53:44,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:44,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:44,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:53:44,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:44,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:44,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:53:44,044 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:53:44,044 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:53:44,044 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:53:44,044 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:53:44,044 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:53:44,044 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:53:44,048 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:44,048 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:44,048 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:44,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:44,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:44,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:53:44,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:53:44,048 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:53:44,048 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:53:44,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:53:44,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:53:44,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:44,049 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:44,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:44,051 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:44,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42342 deadline: 1681937684037, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:44,051 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:44,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42350 deadline: 1681937684037, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:44,053 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:44,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:53:44,052 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/f02035798baf4d76a1a8ef590e665f2e, entries=150, sequenceid=449, filesize=11.9 K 2023-04-19 20:53:44,056 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:44,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42384 deadline: 1681937684051, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:44,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:44,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:44,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:53:44,058 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/C/14d26d1a0f2749e18f2758e1568b64fb as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/14d26d1a0f2749e18f2758e1568b64fb 2023-04-19 20:53:44,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:44,065 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:53:44,065 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:44,065 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:53:44,065 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:44,065 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:53:44,066 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:53:44,065 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:53:44,066 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:44,066 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:44,066 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:44,066 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:44,066 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:44,066 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:53:44,065 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:53:44,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:53:44,066 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:53:44,066 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:53:44,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:53:44,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:53:44,116 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/14d26d1a0f2749e18f2758e1568b64fb, entries=150, sequenceid=449, filesize=11.9 K 2023-04-19 20:53:44,127 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~259.22 KB/265440, currentSize=93.93 KB/96180 for f70bfe945d0054ce5a228cf255d902f5 in 841ms, sequenceid=449, compaction requested=false 2023-04-19 20:53:44,130 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for f70bfe945d0054ce5a228cf255d902f5: 2023-04-19 20:53:44,168 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:53:44,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:53:44,181 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:53:44,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:53:44,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:53:44,192 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:53:44,197 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:53:44,205 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:53:44,205 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:53:44,211 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:44,211 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:44,213 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:53:44,213 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:53:44,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:53:44,221 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:44,222 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:44,222 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:53:44,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:44,223 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:44,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:53:44,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:44,224 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:53:44,232 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:53:44,232 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:53:44,232 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:53:44,232 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:53:44,233 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:53:44,234 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:53:44,234 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:53:44,235 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. 2023-04-19 20:53:44,235 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. started... 2023-04-19 20:53:44,236 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing f70bfe945d0054ce5a228cf255d902f5 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-19 20:53:44,236 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f70bfe945d0054ce5a228cf255d902f5, store=A 2023-04-19 20:53:44,236 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:53:44,236 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f70bfe945d0054ce5a228cf255d902f5, store=B 2023-04-19 20:53:44,236 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:53:44,236 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f70bfe945d0054ce5a228cf255d902f5, store=C 2023-04-19 20:53:44,236 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:53:44,373 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=466 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/A/30020030f05841caa3cd673c2421ca53 2023-04-19 20:53:44,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(9158): Flush requested on f70bfe945d0054ce5a228cf255d902f5 2023-04-19 20:53:44,543 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. as already flushing 2023-04-19 20:53:44,568 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:44,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42350 deadline: 1681937684561, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:44,569 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:44,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42366 deadline: 1681937684563, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:44,572 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:44,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42384 deadline: 1681937684568, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:44,573 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:44,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42380 deadline: 1681937684569, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:44,574 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:44,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42342 deadline: 1681937684570, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:44,674 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:44,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42350 deadline: 1681937684671, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:44,678 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:44,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42366 deadline: 1681937684674, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:44,678 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:44,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42380 deadline: 1681937684677, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:44,679 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:44,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42342 deadline: 1681937684678, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:44,722 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-19 20:53:44,851 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=466 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/B/046686358c124f62a93872cd1f9af3fc 2023-04-19 20:53:44,860 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:meta' 2023-04-19 20:53:44,861 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:namespace' 2023-04-19 20:53:44,884 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:44,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42350 deadline: 1681937684879, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:44,886 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:44,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42380 deadline: 1681937684882, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:44,887 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:44,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42366 deadline: 1681937684882, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:44,888 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:44,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42342 deadline: 1681937684883, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:44,918 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=466 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/C/5c18a7d2e1864b22ba9545a0f8bb6b8a 2023-04-19 20:53:44,931 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/A/30020030f05841caa3cd673c2421ca53 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/30020030f05841caa3cd673c2421ca53 2023-04-19 20:53:44,947 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/30020030f05841caa3cd673c2421ca53, entries=150, sequenceid=466, filesize=11.9 K 2023-04-19 20:53:44,950 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/B/046686358c124f62a93872cd1f9af3fc as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/046686358c124f62a93872cd1f9af3fc 2023-04-19 20:53:44,961 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/046686358c124f62a93872cd1f9af3fc, entries=150, sequenceid=466, filesize=11.9 K 2023-04-19 20:53:44,963 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/C/5c18a7d2e1864b22ba9545a0f8bb6b8a as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/5c18a7d2e1864b22ba9545a0f8bb6b8a 2023-04-19 20:53:45,013 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/5c18a7d2e1864b22ba9545a0f8bb6b8a, entries=150, sequenceid=466, filesize=11.9 K 2023-04-19 20:53:45,016 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=87.22 KB/89310 for f70bfe945d0054ce5a228cf255d902f5 in 780ms, sequenceid=466, compaction requested=true 2023-04-19 20:53:45,016 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f70bfe945d0054ce5a228cf255d902f5: 2023-04-19 20:53:45,016 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. 2023-04-19 20:53:45,016 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:53:45,019 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:53:45,019 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:45,019 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:53:45,019 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:53:45,021 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:45,021 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:45,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:45,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:53:45,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:53:45,022 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:45,022 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:53:45,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:53:45,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:53:45,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:45,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:45,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:53:45,025 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:53:45,025 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5fc00781[Count = 0] remaining members to acquire global barrier 2023-04-19 20:53:45,025 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:53:45,025 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:45,028 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:45,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:45,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:45,028 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:53:45,028 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:53:45,028 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:53:45,029 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:45,029 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:53:45,030 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:53:45,030 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:53:45,030 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:53:45,031 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:45,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:45,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:53:45,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:53:45,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:53:45,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:53:45,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:45,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:45,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:53:45,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:45,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:45,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:53:45,037 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:53:45,037 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:53:45,037 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:53:45,037 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:53:45,037 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:53:45,037 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:53:45,039 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:45,040 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:53:45,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:45,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:45,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:53:45,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:53:45,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:45,042 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:45,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:45,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:53:45,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:53:45,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:53:45,043 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:45,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:45,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:53:45,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:45,046 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:45,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:45,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:53:45,053 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:53:45,053 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:53:45,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:53:45,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:53:45,054 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:45,054 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:45,054 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:45,054 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:53:45,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:53:45,054 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:53:45,054 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:53:45,054 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:45,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:53:45,054 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:45,055 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:45,055 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:45,055 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:53:45,055 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:53:45,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:45,059 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:53:45,155 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:53:45,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:53:45,162 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:53:45,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:53:45,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:53:45,165 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:53:45,165 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:53:45,167 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:53:45,167 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:53:45,167 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:45,167 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:45,169 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:53:45,169 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:45,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:53:45,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:53:45,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:45,170 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:45,170 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:53:45,170 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:45,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:53:45,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:45,171 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:53:45,172 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:53:45,172 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:53:45,172 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:53:45,172 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:53:45,177 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:53:45,178 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:53:45,178 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:53:45,178 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. 2023-04-19 20:53:45,179 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. started... 2023-04-19 20:53:45,179 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing f70bfe945d0054ce5a228cf255d902f5 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-19 20:53:45,179 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f70bfe945d0054ce5a228cf255d902f5, store=A 2023-04-19 20:53:45,181 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:53:45,181 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f70bfe945d0054ce5a228cf255d902f5, store=B 2023-04-19 20:53:45,181 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:53:45,182 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f70bfe945d0054ce5a228cf255d902f5, store=C 2023-04-19 20:53:45,182 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:53:45,189 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. as already flushing 2023-04-19 20:53:45,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(9158): Flush requested on f70bfe945d0054ce5a228cf255d902f5 2023-04-19 20:53:45,231 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:45,239 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:45,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42350 deadline: 1681937685224, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:45,240 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:45,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42366 deadline: 1681937685227, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:45,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42380 deadline: 1681937685218, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:45,243 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:45,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42342 deadline: 1681937685234, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:45,345 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:45,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42350 deadline: 1681937685341, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:45,354 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:45,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42366 deadline: 1681937685350, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:45,355 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:45,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42380 deadline: 1681937685352, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:45,356 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:45,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42342 deadline: 1681937685352, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:45,548 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:45,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42350 deadline: 1681937685548, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:45,559 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:45,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42342 deadline: 1681937685558, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:45,560 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:45,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42380 deadline: 1681937685559, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:45,561 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:45,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42366 deadline: 1681937685560, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:45,585 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:45,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42384 deadline: 1681937685582, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:45,671 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=482 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/A/f08f567cfb31466880001902682269e7 2023-04-19 20:53:45,735 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=482 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/B/98d3753be8bf4318b1a507e913e98d3a 2023-04-19 20:53:45,859 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:45,874 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:45,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42350 deadline: 1681937685852, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:45,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42380 deadline: 1681937685864, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:45,877 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:45,877 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:45,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42342 deadline: 1681937685872, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:45,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42366 deadline: 1681937685873, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:45,900 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(130): Registering adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-04-19 20:53:45,901 INFO [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(134): Registering RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint Metrics about HBase RegionObservers 2023-04-19 20:53:46,188 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=482 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/C/be7f3614a1194d0cbf874797b1d736cd 2023-04-19 20:53:46,210 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/A/f08f567cfb31466880001902682269e7 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/f08f567cfb31466880001902682269e7 2023-04-19 20:53:46,231 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/f08f567cfb31466880001902682269e7, entries=150, sequenceid=482, filesize=11.9 K 2023-04-19 20:53:46,235 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/B/98d3753be8bf4318b1a507e913e98d3a as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/98d3753be8bf4318b1a507e913e98d3a 2023-04-19 20:53:46,255 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/98d3753be8bf4318b1a507e913e98d3a, entries=150, sequenceid=482, filesize=11.9 K 2023-04-19 20:53:46,261 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/C/be7f3614a1194d0cbf874797b1d736cd as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/be7f3614a1194d0cbf874797b1d736cd 2023-04-19 20:53:46,276 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/be7f3614a1194d0cbf874797b1d736cd, entries=150, sequenceid=482, filesize=11.9 K 2023-04-19 20:53:46,278 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for f70bfe945d0054ce5a228cf255d902f5 in 1099ms, sequenceid=482, compaction requested=true 2023-04-19 20:53:46,278 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f70bfe945d0054ce5a228cf255d902f5: 2023-04-19 20:53:46,278 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. 2023-04-19 20:53:46,279 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:53:46,283 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:53:46,283 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:46,283 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:53:46,283 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:53:46,286 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:46,286 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:46,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:46,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:53:46,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:53:46,287 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:46,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:53:46,287 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:53:46,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:53:46,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:46,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:46,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:53:46,293 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:53:46,293 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@ac6ca63[Count = 0] remaining members to acquire global barrier 2023-04-19 20:53:46,293 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:53:46,293 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:46,295 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:46,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:46,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:46,295 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:53:46,296 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:53:46,296 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:53:46,297 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:46,297 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:53:46,300 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:53:46,300 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:53:46,300 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:53:46,324 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:46,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:46,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:53:46,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:53:46,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:53:46,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:53:46,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:46,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:46,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:53:46,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:46,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:46,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:53:46,330 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:53:46,330 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:53:46,330 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:53:46,330 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:53:46,330 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:53:46,330 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:53:46,334 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:46,334 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:53:46,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:46,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:46,334 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:46,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:46,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:53:46,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:53:46,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:53:46,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:53:46,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:53:46,337 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:46,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:46,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:53:46,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:46,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:46,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:46,339 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:46,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:53:46,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:46,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:46,343 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:46,343 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:46,343 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:46,343 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:53:46,343 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:53:46,343 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:53:46,343 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:53:46,343 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:53:46,343 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:53:46,343 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:46,343 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:53:46,343 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:46,344 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:46,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:53:46,343 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:53:46,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:53:46,345 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:46,345 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:53:46,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:53:46,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(9158): Flush requested on f70bfe945d0054ce5a228cf255d902f5 2023-04-19 20:53:46,385 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing f70bfe945d0054ce5a228cf255d902f5 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-19 20:53:46,388 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f70bfe945d0054ce5a228cf255d902f5, store=A 2023-04-19 20:53:46,388 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:53:46,388 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f70bfe945d0054ce5a228cf255d902f5, store=B 2023-04-19 20:53:46,388 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:53:46,388 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f70bfe945d0054ce5a228cf255d902f5, store=C 2023-04-19 20:53:46,388 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:53:46,412 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:46,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42380 deadline: 1681937686402, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:46,412 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:46,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42366 deadline: 1681937686404, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:46,413 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:46,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42342 deadline: 1681937686412, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:46,421 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:46,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42350 deadline: 1681937686412, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:46,443 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:53:46,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:53:46,449 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:53:46,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:53:46,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:53:46,451 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:53:46,451 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:53:46,452 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:53:46,452 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:53:46,452 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:46,453 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:46,454 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:53:46,454 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:46,455 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:53:46,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:53:46,455 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:46,455 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:53:46,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:46,457 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:46,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:53:46,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:46,458 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:53:46,468 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:53:46,468 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:53:46,468 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:53:46,468 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:53:46,472 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:53:46,480 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:53:46,480 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. 2023-04-19 20:53:46,480 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:53:46,480 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. started... 2023-04-19 20:53:46,480 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. as already flushing 2023-04-19 20:53:46,480 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. 2023-04-19 20:53:46,480 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:53:46,480 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:53:46,480 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:46,481 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:53:46,481 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:53:46,483 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:46,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:46,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:53:46,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:53:46,483 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:46,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:53:46,484 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:46,484 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:53:46,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:53:46,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:46,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:46,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:53:46,488 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:53:46,488 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1ba4b348[Count = 0] remaining members to acquire global barrier 2023-04-19 20:53:46,488 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:53:46,488 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:46,492 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:46,492 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:46,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:46,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:46,492 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:53:46,492 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:53:46,492 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:53:46,492 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:53:46,495 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:46,495 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:53:46,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:46,495 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:53:46,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:53:46,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:53:46,495 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:53:46,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:53:46,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:53:46,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:46,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:46,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:53:46,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:46,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:46,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:53:46,506 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:53:46,506 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:53:46,506 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:53:46,506 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:53:46,506 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:53:46,506 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:53:46,508 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:46,508 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:46,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:46,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:46,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:53:46,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:53:46,508 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:53:46,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:46,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:53:46,509 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:53:46,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:53:46,509 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:46,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:46,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:46,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:53:46,511 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:46,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:46,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:46,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:53:46,515 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:46,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42380 deadline: 1681937686514, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:46,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:46,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:46,519 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:53:46,519 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:46,519 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:46,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42366 deadline: 1681937686517, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:46,519 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:53:46,519 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:53:46,519 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:53:46,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:53:46,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:53:46,519 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:53:46,525 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:53:46,526 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:53:46,519 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:46,526 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:46,526 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:46,526 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:46,526 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:46,526 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:46,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:53:46,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:53:46,527 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:53:46,530 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:46,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42342 deadline: 1681937686526, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:46,530 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:46,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42350 deadline: 1681937686526, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:46,626 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:53:46,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:53:46,630 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:53:46,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:53:46,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:53:46,632 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:53:46,633 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:53:46,634 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:53:46,634 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:53:46,634 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:46,634 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:46,637 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:46,637 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:53:46,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:53:46,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:53:46,637 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:46,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:46,637 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:53:46,638 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:46,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:53:46,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:46,638 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:53:46,638 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:53:46,638 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:53:46,638 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:53:46,639 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:53:46,639 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:53:46,640 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:53:46,640 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:53:46,640 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. 2023-04-19 20:53:46,640 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. started... 2023-04-19 20:53:46,640 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. as already flushing 2023-04-19 20:53:46,640 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. 2023-04-19 20:53:46,640 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:53:46,640 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:53:46,640 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:46,640 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:53:46,640 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:53:46,642 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:46,642 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:46,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:46,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:53:46,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:53:46,643 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:46,643 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:53:46,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:53:46,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:53:46,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:46,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:46,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:53:46,646 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:53:46,646 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@626dcbee[Count = 0] remaining members to acquire global barrier 2023-04-19 20:53:46,646 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:53:46,646 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:46,649 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:46,649 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:46,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:46,649 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:53:46,649 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:53:46,649 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:53:46,649 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:46,649 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:53:46,651 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:53:46,651 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:46,651 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:53:46,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:46,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:53:46,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:53:46,651 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:53:46,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:53:46,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:53:46,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:46,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:46,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:53:46,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:46,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:46,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:53:46,655 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:53:46,655 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:53:46,655 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:53:46,655 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:53:46,655 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:53:46,655 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:53:46,658 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:46,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:46,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:53:46,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:53:46,659 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:46,659 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:53:46,659 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:46,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:46,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:53:46,659 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:53:46,659 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:46,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:46,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:53:46,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:53:46,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:46,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:46,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:46,662 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:46,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:53:46,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:46,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:46,669 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:53:46,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:53:46,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:53:46,669 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:53:46,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:53:46,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:53:46,670 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:46,670 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:46,670 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:46,670 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:53:46,670 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:53:46,670 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:46,670 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:46,670 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:46,670 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:46,670 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:53:46,671 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:53:46,670 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:53:46,719 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:46,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42380 deadline: 1681937686718, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:46,729 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:46,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42366 deadline: 1681937686728, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:46,732 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:46,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42342 deadline: 1681937686731, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:46,735 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:46,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42350 deadline: 1681937686733, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:46,771 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:53:46,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:53:46,774 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:53:46,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:53:46,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:53:46,776 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:53:46,776 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:53:46,777 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:53:46,777 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:53:46,778 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:46,778 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:46,779 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:53:46,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:53:46,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:53:46,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:46,781 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:46,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:53:46,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:46,782 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:53:46,782 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:53:46,782 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:53:46,784 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:53:46,784 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:46,784 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:53:46,785 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:53:46,785 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:53:46,786 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:53:46,786 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. 2023-04-19 20:53:46,786 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:46,786 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:53:46,786 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. started... 2023-04-19 20:53:46,786 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. as already flushing 2023-04-19 20:53:46,786 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. 2023-04-19 20:53:46,786 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:53:46,786 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:53:46,786 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:46,786 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:53:46,786 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:53:46,789 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:46,789 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:46,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:46,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:53:46,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:53:46,790 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:46,790 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:53:46,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:53:46,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:53:46,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:46,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:46,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:53:46,792 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:53:46,792 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:53:46,792 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3dce342d[Count = 0] remaining members to acquire global barrier 2023-04-19 20:53:46,792 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:46,794 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:46,794 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:46,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:46,794 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:53:46,794 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:46,794 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:53:46,794 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:53:46,794 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:53:46,796 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:46,796 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:53:46,796 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:53:46,797 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:53:46,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:46,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:53:46,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:53:46,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:53:46,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:53:46,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:46,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:46,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:53:46,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:46,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:46,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:53:46,801 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:53:46,801 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:53:46,801 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:53:46,801 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:53:46,801 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:53:46,801 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:53:46,803 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:46,803 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:53:46,803 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:46,803 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:46,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:46,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:46,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:53:46,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:53:46,804 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:53:46,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:53:46,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:53:46,804 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:46,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:46,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:46,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:53:46,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:46,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:46,806 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:46,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:53:46,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:46,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:46,810 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:53:46,810 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:46,810 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:53:46,810 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:46,810 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:53:46,810 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:53:46,810 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:53:46,810 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:46,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:53:46,811 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:46,811 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:46,811 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:46,811 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:46,811 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:53:46,813 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:53:46,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:53:46,813 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:53:46,813 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:53:46,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:53:46,832 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=501 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/A/310510f59e1944fd9fd6b7481cb2cbed 2023-04-19 20:53:46,868 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=501 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/B/cc0c912afae348bc9eabbc3730a4e297 2023-04-19 20:53:46,913 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:53:46,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:53:46,917 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:53:46,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:53:46,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:53:46,919 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:53:46,919 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:53:46,920 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:53:46,920 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:53:46,920 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:46,920 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:46,922 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:46,922 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:53:46,922 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:53:46,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:53:46,922 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:46,922 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:53:46,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:46,923 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:46,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:53:46,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:46,924 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:53:46,925 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:53:46,925 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:53:46,925 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:53:46,925 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:53:46,927 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:53:46,928 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:53:46,928 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:53:46,928 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. 2023-04-19 20:53:46,928 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. started... 2023-04-19 20:53:46,928 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. as already flushing 2023-04-19 20:53:46,928 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. 2023-04-19 20:53:46,930 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:53:46,930 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:53:46,930 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:46,930 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:53:46,930 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:53:46,933 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:46,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:46,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:53:46,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:53:46,933 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:46,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:53:46,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:53:46,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:46,934 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:46,935 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:53:46,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:46,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:53:46,936 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:53:46,936 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e77425[Count = 0] remaining members to acquire global barrier 2023-04-19 20:53:46,936 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:53:46,936 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:46,939 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:46,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:46,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:46,940 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:53:46,940 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:53:46,940 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:53:46,943 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:46,943 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:53:46,945 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:46,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:46,945 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:53:46,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:53:46,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:53:46,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:53:46,946 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:53:46,952 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:53:46,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:53:46,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:46,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:46,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:53:46,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:46,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:46,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:53:46,968 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:53:46,968 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:53:46,968 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:53:46,968 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:53:46,968 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:53:46,969 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:53:46,971 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:46,971 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:53:46,971 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:46,971 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:46,972 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:46,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:46,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:53:46,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:53:46,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:46,972 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:53:46,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:53:46,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:53:46,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:46,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:46,973 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:46,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:53:46,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:46,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:46,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:53:46,978 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:53:46,978 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:53:46,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:53:46,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:53:46,979 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:46,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:53:46,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:53:46,979 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:53:46,979 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:46,979 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:46,979 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:46,979 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:46,979 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:53:46,979 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:53:46,980 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:53:46,979 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:46,980 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:46,980 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:53:46,980 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:53:47,027 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:47,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42380 deadline: 1681937687023, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:47,033 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:47,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42366 deadline: 1681937687031, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:47,036 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:47,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42342 deadline: 1681937687034, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:47,042 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:47,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42350 deadline: 1681937687038, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:47,080 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:53:47,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:53:47,085 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:53:47,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:53:47,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:53:47,089 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:53:47,089 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:53:47,090 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:53:47,090 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:53:47,091 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:47,091 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:47,092 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:47,092 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:53:47,092 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:53:47,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:53:47,093 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:47,111 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:53:47,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:47,111 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:47,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:53:47,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:47,112 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:53:47,113 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:53:47,113 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:53:47,113 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:53:47,115 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:53:47,115 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:53:47,115 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:53:47,115 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. 2023-04-19 20:53:47,115 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. started... 2023-04-19 20:53:47,115 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. as already flushing 2023-04-19 20:53:47,115 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. 2023-04-19 20:53:47,115 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:53:47,115 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:53:47,115 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:47,116 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:53:47,116 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:53:47,118 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:47,118 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:47,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:47,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:53:47,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:53:47,119 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:47,119 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:53:47,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:53:47,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:53:47,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:47,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:47,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:53:47,120 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:53:47,120 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3fe7a9a2[Count = 0] remaining members to acquire global barrier 2023-04-19 20:53:47,122 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:53:47,122 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:47,123 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:47,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:47,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:47,123 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:53:47,124 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:53:47,124 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:53:47,124 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:47,124 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:53:47,125 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:53:47,125 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:47,125 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:53:47,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:47,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:53:47,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:53:47,125 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:53:47,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:53:47,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:53:47,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:47,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:47,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:53:47,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:47,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:47,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:53:47,133 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:53:47,133 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:53:47,133 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:53:47,133 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:53:47,133 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:53:47,133 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:53:47,135 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:47,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:47,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:53:47,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:53:47,136 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:47,136 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:53:47,136 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:47,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:47,136 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:53:47,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:53:47,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:53:47,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:47,137 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:47,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:47,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:53:47,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:47,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:47,139 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:47,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:53:47,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:47,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:47,143 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:47,143 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:53:47,143 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:47,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:53:47,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:53:47,143 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:53:47,143 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:47,143 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:53:47,143 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:47,143 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:47,144 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:47,145 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:47,145 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:53:47,145 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:53:47,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:53:47,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:53:47,145 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:53:47,145 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:53:47,245 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:53:47,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:53:47,249 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:53:47,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:53:47,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:53:47,251 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:53:47,251 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:53:47,256 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:53:47,256 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:53:47,256 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:47,256 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:47,258 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:47,258 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:53:47,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:53:47,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:53:47,259 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:47,259 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:53:47,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:47,260 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:47,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:53:47,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:47,261 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:53:47,261 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:53:47,261 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:53:47,261 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:53:47,261 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:53:47,262 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:53:47,263 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:53:47,263 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:53:47,263 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. 2023-04-19 20:53:47,263 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. started... 2023-04-19 20:53:47,263 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. as already flushing 2023-04-19 20:53:47,263 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. 2023-04-19 20:53:47,263 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:53:47,263 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:53:47,263 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:47,263 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:53:47,263 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:53:47,267 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:47,267 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:47,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:47,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:53:47,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:53:47,267 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:47,268 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:53:47,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:53:47,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:53:47,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:47,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:47,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:53:47,270 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:53:47,270 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@33d9370b[Count = 0] remaining members to acquire global barrier 2023-04-19 20:53:47,270 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:53:47,270 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:47,273 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:47,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:47,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:47,273 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:53:47,273 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:53:47,273 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:47,273 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:53:47,273 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:53:47,274 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:53:47,274 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:47,274 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:53:47,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:47,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:53:47,275 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:53:47,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:53:47,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:53:47,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:53:47,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:47,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:47,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:53:47,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:47,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:47,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:53:47,279 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:53:47,279 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:53:47,279 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:53:47,279 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:53:47,279 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:53:47,279 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:53:47,281 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:47,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:47,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:53:47,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:53:47,281 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:47,282 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:53:47,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:47,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:47,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:53:47,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:53:47,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:53:47,282 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:47,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:47,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:47,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:53:47,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:47,284 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:47,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:47,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:53:47,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:47,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:47,287 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:53:47,287 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:47,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:53:47,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:53:47,287 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:53:47,287 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:53:47,287 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:47,287 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:47,287 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:47,287 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:47,287 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:47,288 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:47,287 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:53:47,287 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:53:47,288 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:53:47,289 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:53:47,289 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:53:47,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:53:47,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:53:47,298 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=501 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/C/78ae43b6947e4c41a8a8fe9c8f556631 2023-04-19 20:53:47,309 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/A/310510f59e1944fd9fd6b7481cb2cbed as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/310510f59e1944fd9fd6b7481cb2cbed 2023-04-19 20:53:47,318 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/310510f59e1944fd9fd6b7481cb2cbed, entries=200, sequenceid=501, filesize=14.3 K 2023-04-19 20:53:47,322 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/B/cc0c912afae348bc9eabbc3730a4e297 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/cc0c912afae348bc9eabbc3730a4e297 2023-04-19 20:53:47,337 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/cc0c912afae348bc9eabbc3730a4e297, entries=150, sequenceid=501, filesize=11.9 K 2023-04-19 20:53:47,340 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/C/78ae43b6947e4c41a8a8fe9c8f556631 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/78ae43b6947e4c41a8a8fe9c8f556631 2023-04-19 20:53:47,357 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/78ae43b6947e4c41a8a8fe9c8f556631, entries=150, sequenceid=501, filesize=11.9 K 2023-04-19 20:53:47,360 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=73.80 KB/75570 for f70bfe945d0054ce5a228cf255d902f5 in 975ms, sequenceid=501, compaction requested=true 2023-04-19 20:53:47,360 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for f70bfe945d0054ce5a228cf255d902f5: 2023-04-19 20:53:47,367 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 20:53:47,367 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 20:53:47,367 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 20:53:47,367 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-19 20:53:47,367 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-19 20:53:47,375 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 58286 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-19 20:53:47,377 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 75370 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-19 20:53:47,377 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1912): f70bfe945d0054ce5a228cf255d902f5/A is initiating minor compaction (all files) 2023-04-19 20:53:47,377 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1912): f70bfe945d0054ce5a228cf255d902f5/B is initiating minor compaction (all files) 2023-04-19 20:53:47,377 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of f70bfe945d0054ce5a228cf255d902f5/A in TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. 2023-04-19 20:53:47,377 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2259): Starting compaction of f70bfe945d0054ce5a228cf255d902f5/B in TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. 2023-04-19 20:53:47,378 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/c5820d14b7d445a39e49141037c74f93, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/6d7e233041aa47a8a61054b74f3c78e3, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/30020030f05841caa3cd673c2421ca53, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/f08f567cfb31466880001902682269e7, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/310510f59e1944fd9fd6b7481cb2cbed] into tmpdir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp, totalSize=73.6 K 2023-04-19 20:53:47,378 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/daf9346af1d1405c96ef9a592f36872e, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/f02035798baf4d76a1a8ef590e665f2e, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/046686358c124f62a93872cd1f9af3fc, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/98d3753be8bf4318b1a507e913e98d3a, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/cc0c912afae348bc9eabbc3730a4e297] into tmpdir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp, totalSize=56.9 K 2023-04-19 20:53:47,381 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting c5820d14b7d445a39e49141037c74f93, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=7, earliestPutTs=1681937621930 2023-04-19 20:53:47,383 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting daf9346af1d1405c96ef9a592f36872e, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=7, earliestPutTs=1681937621930 2023-04-19 20:53:47,385 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 6d7e233041aa47a8a61054b74f3c78e3, keycount=450, bloomtype=ROW, size=26.2 K, encoding=NONE, compression=NONE, seqNum=449, earliestPutTs=1681937623182 2023-04-19 20:53:47,386 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting f02035798baf4d76a1a8ef590e665f2e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=449, earliestPutTs=1681937623251 2023-04-19 20:53:47,386 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 30020030f05841caa3cd673c2421ca53, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=466, earliestPutTs=1681937623318 2023-04-19 20:53:47,387 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 046686358c124f62a93872cd1f9af3fc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=466, earliestPutTs=1681937623318 2023-04-19 20:53:47,387 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting f08f567cfb31466880001902682269e7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=482, earliestPutTs=1681937624560 2023-04-19 20:53:47,387 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 98d3753be8bf4318b1a507e913e98d3a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=482, earliestPutTs=1681937624560 2023-04-19 20:53:47,388 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 310510f59e1944fd9fd6b7481cb2cbed, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=501, earliestPutTs=1681937625212 2023-04-19 20:53:47,389 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:53:47,390 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting cc0c912afae348bc9eabbc3730a4e297, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=501, earliestPutTs=1681937625224 2023-04-19 20:53:47,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:53:47,397 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:53:47,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:53:47,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:53:47,399 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:53:47,399 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:53:47,400 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:53:47,401 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:53:47,401 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:47,401 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:47,403 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:53:47,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:53:47,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:53:47,403 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:47,403 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:47,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:47,403 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:53:47,404 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:47,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:53:47,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:47,405 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:53:47,406 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:53:47,406 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:53:47,406 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:53:47,406 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:53:47,408 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:53:47,408 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:53:47,408 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:53:47,408 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. 2023-04-19 20:53:47,408 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. started... 2023-04-19 20:53:47,409 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing f70bfe945d0054ce5a228cf255d902f5 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-19 20:53:47,409 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f70bfe945d0054ce5a228cf255d902f5, store=A 2023-04-19 20:53:47,409 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:53:47,409 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f70bfe945d0054ce5a228cf255d902f5, store=B 2023-04-19 20:53:47,410 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:53:47,410 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f70bfe945d0054ce5a228cf255d902f5, store=C 2023-04-19 20:53:47,410 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:53:47,445 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=515 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/A/f17c859230b84ccb8b0da865f1285cf2 2023-04-19 20:53:47,446 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] throttle.PressureAwareThroughputController(145): f70bfe945d0054ce5a228cf255d902f5#A#compaction#17 average throughput is 0.44 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-19 20:53:47,451 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] throttle.PressureAwareThroughputController(145): f70bfe945d0054ce5a228cf255d902f5#B#compaction#16 average throughput is 0.30 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 20:53:47,518 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=515 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/B/e158ae9bbf07487b8bdb88909892fd88 2023-04-19 20:53:47,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(9158): Flush requested on f70bfe945d0054ce5a228cf255d902f5 2023-04-19 20:53:47,533 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. as already flushing 2023-04-19 20:53:47,559 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=515 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/C/d397c066eb50460e97d2137e50678dc5 2023-04-19 20:53:47,571 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/A/f17c859230b84ccb8b0da865f1285cf2 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/f17c859230b84ccb8b0da865f1285cf2 2023-04-19 20:53:47,582 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/f17c859230b84ccb8b0da865f1285cf2, entries=150, sequenceid=515, filesize=11.9 K 2023-04-19 20:53:47,583 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/B/e158ae9bbf07487b8bdb88909892fd88 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/e158ae9bbf07487b8bdb88909892fd88 2023-04-19 20:53:47,589 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:47,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42342 deadline: 1681937687577, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:47,599 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:47,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42380 deadline: 1681937687578, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:47,600 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:47,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42350 deadline: 1681937687579, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:47,601 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:47,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42366 deadline: 1681937687580, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:47,601 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:47,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42384 deadline: 1681937687597, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:47,604 DEBUG [Thread-141] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4227 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5., hostname=jenkins-hbase4.apache.org,35247,1681937616971, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-19 20:53:47,606 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/e158ae9bbf07487b8bdb88909892fd88, entries=150, sequenceid=515, filesize=11.9 K 2023-04-19 20:53:47,607 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/C/d397c066eb50460e97d2137e50678dc5 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/d397c066eb50460e97d2137e50678dc5 2023-04-19 20:53:47,633 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/d397c066eb50460e97d2137e50678dc5, entries=150, sequenceid=515, filesize=11.9 K 2023-04-19 20:53:47,634 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for f70bfe945d0054ce5a228cf255d902f5 in 225ms, sequenceid=515, compaction requested=true 2023-04-19 20:53:47,634 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f70bfe945d0054ce5a228cf255d902f5: 2023-04-19 20:53:47,634 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. 2023-04-19 20:53:47,634 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:53:47,635 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:53:47,635 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:47,635 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:53:47,635 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:53:47,638 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:47,638 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:47,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:47,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:53:47,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:53:47,638 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:47,638 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:53:47,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:53:47,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:53:47,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:47,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:47,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:53:47,642 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:53:47,642 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ffa1201[Count = 0] remaining members to acquire global barrier 2023-04-19 20:53:47,642 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:53:47,642 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:47,645 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:47,645 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:47,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:47,645 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:53:47,645 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:53:47,645 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:53:47,646 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:47,646 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:53:47,647 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:47,647 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:53:47,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:47,647 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:53:47,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:53:47,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:53:47,647 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:53:47,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:53:47,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:53:47,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:47,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:47,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:53:47,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:47,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:47,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:53:47,651 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:53:47,651 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:53:47,651 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:53:47,651 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:53:47,651 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:53:47,651 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:53:47,653 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:47,653 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:47,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:47,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:47,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:53:47,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:53:47,653 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:53:47,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:47,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:53:47,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:53:47,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:53:47,654 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:47,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:47,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:47,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:53:47,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:47,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:47,657 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:47,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:53:47,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:47,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:47,660 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:47,660 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:47,660 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:53:47,660 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:53:47,660 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:47,661 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:53:47,661 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:53:47,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:53:47,660 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:53:47,661 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:53:47,661 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:47,661 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:47,662 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:47,662 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:47,662 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:53:47,662 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:53:47,662 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:53:47,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:53:47,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(9158): Flush requested on f70bfe945d0054ce5a228cf255d902f5 2023-04-19 20:53:47,698 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing f70bfe945d0054ce5a228cf255d902f5 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-19 20:53:47,699 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f70bfe945d0054ce5a228cf255d902f5, store=A 2023-04-19 20:53:47,699 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:53:47,699 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f70bfe945d0054ce5a228cf255d902f5, store=B 2023-04-19 20:53:47,699 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:53:47,699 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f70bfe945d0054ce5a228cf255d902f5, store=C 2023-04-19 20:53:47,699 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:53:47,729 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:47,729 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:47,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42366 deadline: 1681937687717, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:47,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42380 deadline: 1681937687715, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:47,736 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:47,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42342 deadline: 1681937687729, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:47,737 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:47,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42350 deadline: 1681937687731, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:47,762 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:53:47,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:53:47,768 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:53:47,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:53:47,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:53:47,780 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:53:47,780 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:53:47,785 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:53:47,785 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:53:47,786 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:47,786 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:47,788 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:53:47,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:53:47,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:53:47,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:47,790 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:47,790 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:47,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:53:47,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:47,790 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:53:47,791 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:47,791 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:53:47,791 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:53:47,791 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:53:47,791 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:53:47,791 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:53:47,793 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:53:47,795 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:53:47,795 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:53:47,795 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. 2023-04-19 20:53:47,796 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. started... 2023-04-19 20:53:47,799 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. as already flushing 2023-04-19 20:53:47,799 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. 2023-04-19 20:53:47,799 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:53:47,799 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:53:47,799 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:47,799 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:53:47,799 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:53:47,801 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:47,801 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:47,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:47,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:53:47,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:53:47,802 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:47,802 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:53:47,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:53:47,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:53:47,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:47,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:47,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:53:47,804 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:53:47,804 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@45061787[Count = 0] remaining members to acquire global barrier 2023-04-19 20:53:47,805 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:53:47,805 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:47,807 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:47,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:47,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:47,807 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:53:47,807 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:53:47,808 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:53:47,808 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:47,808 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:53:47,810 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:53:47,810 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:47,810 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:53:47,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:47,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:53:47,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:53:47,811 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:53:47,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:53:47,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:53:47,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:47,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:47,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:53:47,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:47,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:47,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:53:47,815 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:53:47,815 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:53:47,815 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:53:47,815 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:53:47,815 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:53:47,815 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:53:47,818 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:47,818 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:53:47,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:47,818 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:47,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:47,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:47,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:53:47,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:53:47,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:53:47,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:53:47,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:53:47,819 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:47,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:47,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:47,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:53:47,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:47,821 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:47,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:47,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:53:47,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:47,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:47,825 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:53:47,825 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:53:47,825 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:53:47,826 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:47,826 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:53:47,826 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:53:47,826 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:47,826 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:47,826 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:47,826 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:47,826 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:47,826 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:47,828 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:53:47,828 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:53:47,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:53:47,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:53:47,828 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:53:47,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:53:47,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:53:47,832 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:47,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42380 deadline: 1681937687832, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:47,833 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:47,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42366 deadline: 1681937687832, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:47,842 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:47,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42342 deadline: 1681937687839, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:47,843 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:47,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42350 deadline: 1681937687839, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:47,898 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/A/acfc472dda4a49979cd46ad6194df803 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/acfc472dda4a49979cd46ad6194df803 2023-04-19 20:53:47,914 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/B/0cfcc2ef2a8e4ed681330504c0cdc822 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/0cfcc2ef2a8e4ed681330504c0cdc822 2023-04-19 20:53:47,926 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:53:47,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:53:47,931 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in f70bfe945d0054ce5a228cf255d902f5/B of f70bfe945d0054ce5a228cf255d902f5 into 0cfcc2ef2a8e4ed681330504c0cdc822(size=12.1 K), total size for store is 24.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 20:53:47,931 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in f70bfe945d0054ce5a228cf255d902f5/A of f70bfe945d0054ce5a228cf255d902f5 into acfc472dda4a49979cd46ad6194df803(size=12.1 K), total size for store is 24.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 20:53:47,932 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for f70bfe945d0054ce5a228cf255d902f5: 2023-04-19 20:53:47,932 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for f70bfe945d0054ce5a228cf255d902f5: 2023-04-19 20:53:47,932 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5., storeName=f70bfe945d0054ce5a228cf255d902f5/B, priority=11, startTime=1681937627367; duration=0sec 2023-04-19 20:53:47,932 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5., storeName=f70bfe945d0054ce5a228cf255d902f5/A, priority=11, startTime=1681937627362; duration=0sec 2023-04-19 20:53:47,933 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 20:53:47,933 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-19 20:53:47,933 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 20:53:47,933 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:53:47,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:53:47,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:53:47,936 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:53:47,936 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:53:47,937 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:53:47,937 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:53:47,938 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:47,938 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:47,941 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:47,941 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:53:47,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:53:47,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:53:47,941 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:47,941 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:53:47,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:47,943 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 70472 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-19 20:53:47,943 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:47,943 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1912): f70bfe945d0054ce5a228cf255d902f5/C is initiating minor compaction (all files) 2023-04-19 20:53:47,943 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of f70bfe945d0054ce5a228cf255d902f5/C in TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. 2023-04-19 20:53:47,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:53:47,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:47,945 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/0fb95bdc773047e3b2042cbe5f94f3c0, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/14d26d1a0f2749e18f2758e1568b64fb, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/5c18a7d2e1864b22ba9545a0f8bb6b8a, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/be7f3614a1194d0cbf874797b1d736cd, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/78ae43b6947e4c41a8a8fe9c8f556631, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/d397c066eb50460e97d2137e50678dc5] into tmpdir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp, totalSize=68.8 K 2023-04-19 20:53:47,945 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:53:47,947 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:53:47,947 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:53:47,947 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 0fb95bdc773047e3b2042cbe5f94f3c0, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=7, earliestPutTs=1681937621930 2023-04-19 20:53:47,948 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:53:47,950 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:53:47,950 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:53:47,951 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 14d26d1a0f2749e18f2758e1568b64fb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=449, earliestPutTs=1681937623251 2023-04-19 20:53:47,951 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 5c18a7d2e1864b22ba9545a0f8bb6b8a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=466, earliestPutTs=1681937623318 2023-04-19 20:53:47,952 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting be7f3614a1194d0cbf874797b1d736cd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=482, earliestPutTs=1681937624560 2023-04-19 20:53:47,953 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 78ae43b6947e4c41a8a8fe9c8f556631, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=501, earliestPutTs=1681937625224 2023-04-19 20:53:47,953 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting d397c066eb50460e97d2137e50678dc5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=515, earliestPutTs=1681937626400 2023-04-19 20:53:47,954 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:53:47,954 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:53:47,954 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. 2023-04-19 20:53:47,955 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. started... 2023-04-19 20:53:47,955 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. as already flushing 2023-04-19 20:53:47,955 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. 2023-04-19 20:53:47,955 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:53:47,955 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:53:47,955 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:47,955 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:53:47,955 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:53:47,957 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:47,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:47,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:53:47,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:53:47,957 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:47,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:53:47,958 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:47,958 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:53:47,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:53:47,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:47,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:47,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:53:47,960 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:53:47,960 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@472e0c3c[Count = 0] remaining members to acquire global barrier 2023-04-19 20:53:47,960 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:53:47,960 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:47,966 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:47,966 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:53:47,967 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:47,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:47,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:47,967 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:53:47,967 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:53:47,967 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:53:47,969 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:47,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:47,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:53:47,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:53:47,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:53:47,970 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:53:47,970 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:53:47,970 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:53:47,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:53:47,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:47,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:47,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:53:47,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:47,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:47,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:53:47,974 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:53:47,974 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:53:47,974 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:53:47,974 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:53:47,974 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:53:47,975 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:53:47,976 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:47,977 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:53:47,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:47,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:47,977 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:47,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:53:47,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:53:47,977 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:47,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:47,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:53:47,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:53:47,979 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:47,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:47,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:53:47,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:47,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:53:47,980 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] throttle.PressureAwareThroughputController(145): f70bfe945d0054ce5a228cf255d902f5#C#compaction#21 average throughput is 1.09 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 20:53:47,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:47,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:47,984 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:53:47,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:53:47,984 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:47,984 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:53:47,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:53:47,984 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:47,984 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:53:47,984 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:53:47,984 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:47,985 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:47,985 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:47,985 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:47,985 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:47,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:53:47,985 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:53:47,985 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:53:47,984 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:53:47,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:53:47,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:53:47,989 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:53:48,017 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/C/492a54bd908348119b1b63b030bee115 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/492a54bd908348119b1b63b030bee115 2023-04-19 20:53:48,025 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in f70bfe945d0054ce5a228cf255d902f5/C of f70bfe945d0054ce5a228cf255d902f5 into 492a54bd908348119b1b63b030bee115(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 20:53:48,026 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for f70bfe945d0054ce5a228cf255d902f5: 2023-04-19 20:53:48,026 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5., storeName=f70bfe945d0054ce5a228cf255d902f5/C, priority=10, startTime=1681937627367; duration=0sec 2023-04-19 20:53:48,026 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 20:53:48,038 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:48,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42366 deadline: 1681937688036, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:48,038 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:48,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42380 deadline: 1681937688036, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:48,046 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:48,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42342 deadline: 1681937688045, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:48,048 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:48,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42350 deadline: 1681937688046, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:48,085 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:53:48,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:53:48,089 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:53:48,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:53:48,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:53:48,091 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:53:48,091 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:53:48,092 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:53:48,092 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:53:48,093 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:48,093 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:48,094 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:48,094 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:53:48,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:53:48,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:53:48,095 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:48,096 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:53:48,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:48,097 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:48,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:53:48,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:48,098 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:53:48,098 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:53:48,098 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:53:48,098 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:53:48,098 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:53:48,099 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:53:48,100 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:53:48,100 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:53:48,100 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. 2023-04-19 20:53:48,100 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. started... 2023-04-19 20:53:48,100 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. as already flushing 2023-04-19 20:53:48,100 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. 2023-04-19 20:53:48,101 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:53:48,101 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:53:48,101 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:48,101 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:53:48,101 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:53:48,103 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:48,103 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:48,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:48,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:53:48,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:53:48,103 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:48,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:53:48,103 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:53:48,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:53:48,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:48,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:48,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:53:48,108 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:53:48,108 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3d351abd[Count = 0] remaining members to acquire global barrier 2023-04-19 20:53:48,108 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:53:48,108 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:48,111 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:48,111 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:53:48,112 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:48,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:48,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:48,112 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:53:48,112 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:53:48,112 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:53:48,115 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:48,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:48,115 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:53:48,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:53:48,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:53:48,115 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:53:48,115 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:53:48,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:53:48,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:53:48,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:48,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:48,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:53:48,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:48,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:48,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:53:48,123 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:53:48,123 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:53:48,123 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:53:48,123 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:53:48,123 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:53:48,123 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:53:48,126 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:48,126 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:48,126 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:53:48,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:48,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:48,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:48,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:53:48,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:53:48,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:53:48,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:53:48,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:53:48,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:48,127 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:48,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:48,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:53:48,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:48,131 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:48,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:48,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:53:48,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:48,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:48,134 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:48,135 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:48,135 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:48,135 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:53:48,135 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:53:48,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:53:48,135 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:53:48,135 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:53:48,135 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:53:48,135 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:48,140 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:53:48,136 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:53:48,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:53:48,140 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:53:48,140 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:48,140 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:48,140 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:48,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:53:48,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:53:48,155 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=534 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/A/ff28f839fa734083a05196ca36963ce7 2023-04-19 20:53:48,200 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=534 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/B/5c68103d0c30480bac080a8759341ce1 2023-04-19 20:53:48,236 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=534 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/C/5e1159c20c7a468d8890a6be2b8535b1 2023-04-19 20:53:48,240 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:53:48,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:53:48,245 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:53:48,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:53:48,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:53:48,247 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:53:48,247 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:53:48,248 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:53:48,248 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:53:48,249 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:48,249 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:48,250 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/A/ff28f839fa734083a05196ca36963ce7 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/ff28f839fa734083a05196ca36963ce7 2023-04-19 20:53:48,251 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:48,251 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:53:48,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:53:48,251 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:48,251 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:53:48,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:53:48,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:48,252 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:48,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:53:48,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:48,253 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:53:48,253 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:53:48,253 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:53:48,253 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:53:48,253 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:53:48,254 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:53:48,256 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:53:48,256 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:53:48,256 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. 2023-04-19 20:53:48,256 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. started... 2023-04-19 20:53:48,257 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. as already flushing 2023-04-19 20:53:48,257 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. 2023-04-19 20:53:48,257 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:53:48,257 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:53:48,257 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:48,257 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:53:48,257 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:53:48,260 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:48,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:48,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:53:48,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:53:48,261 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/ff28f839fa734083a05196ca36963ce7, entries=200, sequenceid=534, filesize=14.3 K 2023-04-19 20:53:48,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:53:48,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:53:48,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:48,265 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/B/5c68103d0c30480bac080a8759341ce1 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/5c68103d0c30480bac080a8759341ce1 2023-04-19 20:53:48,265 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:48,265 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:48,265 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:53:48,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:48,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:53:48,267 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:53:48,268 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@12707d00[Count = 0] remaining members to acquire global barrier 2023-04-19 20:53:48,268 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:53:48,268 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:48,271 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:48,271 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:48,271 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:48,271 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:53:48,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:48,271 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:53:48,271 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:53:48,271 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:53:48,273 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:53:48,273 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:48,273 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:53:48,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:48,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:53:48,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:53:48,273 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:53:48,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:53:48,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:53:48,276 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/5c68103d0c30480bac080a8759341ce1, entries=150, sequenceid=534, filesize=11.9 K 2023-04-19 20:53:48,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:48,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:48,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:53:48,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:48,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:48,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:53:48,282 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:53:48,283 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:53:48,283 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:53:48,283 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:53:48,283 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:53:48,283 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:53:48,283 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/C/5e1159c20c7a468d8890a6be2b8535b1 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/5e1159c20c7a468d8890a6be2b8535b1 2023-04-19 20:53:48,285 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:48,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:48,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:53:48,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:53:48,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:53:48,286 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:48,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:48,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:53:48,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:48,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:48,288 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:48,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:53:48,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:48,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:48,292 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:48,292 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:48,292 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:48,292 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:53:48,292 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:53:48,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:48,292 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:53:48,292 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:53:48,292 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:53:48,292 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:53:48,292 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:53:48,292 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:48,292 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:53:48,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:48,294 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:48,294 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:48,294 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:48,294 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:48,294 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:53:48,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:53:48,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:53:48,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:53:48,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:53:48,296 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/5e1159c20c7a468d8890a6be2b8535b1, entries=150, sequenceid=534, filesize=11.9 K 2023-04-19 20:53:48,299 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for f70bfe945d0054ce5a228cf255d902f5 in 601ms, sequenceid=534, compaction requested=true 2023-04-19 20:53:48,299 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for f70bfe945d0054ce5a228cf255d902f5: 2023-04-19 20:53:48,299 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 20:53:48,299 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-19 20:53:48,299 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 20:53:48,299 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-19 20:53:48,299 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 20:53:48,303 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39170 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-19 20:53:48,303 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36730 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-19 20:53:48,303 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1912): f70bfe945d0054ce5a228cf255d902f5/A is initiating minor compaction (all files) 2023-04-19 20:53:48,303 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1912): f70bfe945d0054ce5a228cf255d902f5/B is initiating minor compaction (all files) 2023-04-19 20:53:48,303 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2259): Starting compaction of f70bfe945d0054ce5a228cf255d902f5/B in TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. 2023-04-19 20:53:48,303 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/0cfcc2ef2a8e4ed681330504c0cdc822, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/e158ae9bbf07487b8bdb88909892fd88, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/5c68103d0c30480bac080a8759341ce1] into tmpdir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp, totalSize=35.9 K 2023-04-19 20:53:48,303 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of f70bfe945d0054ce5a228cf255d902f5/A in TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. 2023-04-19 20:53:48,304 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/acfc472dda4a49979cd46ad6194df803, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/f17c859230b84ccb8b0da865f1285cf2, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/ff28f839fa734083a05196ca36963ce7] into tmpdir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp, totalSize=38.3 K 2023-04-19 20:53:48,305 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 0cfcc2ef2a8e4ed681330504c0cdc822, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=501, earliestPutTs=1681937625224 2023-04-19 20:53:48,305 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting acfc472dda4a49979cd46ad6194df803, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=501, earliestPutTs=1681937625224 2023-04-19 20:53:48,324 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting e158ae9bbf07487b8bdb88909892fd88, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=515, earliestPutTs=1681937626400 2023-04-19 20:53:48,324 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting f17c859230b84ccb8b0da865f1285cf2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=515, earliestPutTs=1681937626400 2023-04-19 20:53:48,325 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting ff28f839fa734083a05196ca36963ce7, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=534, earliestPutTs=1681937627575 2023-04-19 20:53:48,328 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 5c68103d0c30480bac080a8759341ce1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=534, earliestPutTs=1681937627577 2023-04-19 20:53:48,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(9158): Flush requested on f70bfe945d0054ce5a228cf255d902f5 2023-04-19 20:53:48,343 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing f70bfe945d0054ce5a228cf255d902f5 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-19 20:53:48,344 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f70bfe945d0054ce5a228cf255d902f5, store=A 2023-04-19 20:53:48,344 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:53:48,344 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f70bfe945d0054ce5a228cf255d902f5, store=B 2023-04-19 20:53:48,344 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:53:48,344 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f70bfe945d0054ce5a228cf255d902f5, store=C 2023-04-19 20:53:48,344 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:53:48,350 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] throttle.PressureAwareThroughputController(145): f70bfe945d0054ce5a228cf255d902f5#B#compaction#24 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 20:53:48,368 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] throttle.PressureAwareThroughputController(145): f70bfe945d0054ce5a228cf255d902f5#A#compaction#26 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 20:53:48,402 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:53:48,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:53:48,410 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:53:48,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:53:48,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:53:48,412 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:53:48,415 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:53:48,428 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:53:48,435 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:53:48,437 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:48,437 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:48,439 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:53:48,439 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:48,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:53:48,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:53:48,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:48,440 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:48,441 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:53:48,441 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:48,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:53:48,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:48,442 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:53:48,442 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:53:48,442 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:53:48,442 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:53:48,442 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:53:48,448 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:53:48,448 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:53:48,448 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. 2023-04-19 20:53:48,448 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:53:48,448 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. started... 2023-04-19 20:53:48,448 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. as already flushing 2023-04-19 20:53:48,448 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. 2023-04-19 20:53:48,448 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:53:48,448 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:53:48,448 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:48,449 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:53:48,449 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:53:48,453 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=551 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/A/361fbc254d34492ea61eff57284aec32 2023-04-19 20:53:48,454 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:48,454 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:48,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:48,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:53:48,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:53:48,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:53:48,455 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:48,455 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:53:48,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:53:48,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:48,457 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:48,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42366 deadline: 1681937688438, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:48,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:48,468 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:48,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42342 deadline: 1681937688449, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:48,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:53:48,471 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:53:48,471 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@67bffae6[Count = 0] remaining members to acquire global barrier 2023-04-19 20:53:48,471 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:53:48,471 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:48,473 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:48,473 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:48,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:48,473 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:53:48,474 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:53:48,474 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:53:48,474 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:48,474 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:53:48,476 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:48,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42350 deadline: 1681937688468, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:48,476 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:53:48,476 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:53:48,476 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:53:48,476 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:48,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:48,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:53:48,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:53:48,477 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:48,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:53:48,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42380 deadline: 1681937688469, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:48,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:53:48,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:48,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:48,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:53:48,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:48,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:48,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:53:48,481 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:53:48,481 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:53:48,481 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:53:48,481 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:53:48,481 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:53:48,481 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:53:48,486 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:48,486 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:48,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:48,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:48,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:48,486 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:53:48,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:53:48,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:53:48,500 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:53:48,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:53:48,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:48,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:53:48,502 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:48,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:48,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:53:48,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:48,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:48,503 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:48,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:53:48,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:48,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:48,507 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:53:48,508 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:53:48,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:53:48,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:53:48,508 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:48,508 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:48,508 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:53:48,508 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:53:48,508 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:53:48,508 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:48,508 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:48,508 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:48,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:53:48,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:53:48,508 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:53:48,508 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:53:48,508 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:48,509 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:48,570 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:48,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42342 deadline: 1681937688570, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:48,575 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:48,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42366 deadline: 1681937688574, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:48,579 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:48,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42350 deadline: 1681937688578, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:48,580 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:48,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42380 deadline: 1681937688580, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:48,609 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:53:48,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:53:48,614 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:53:48,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:53:48,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:53:48,617 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:53:48,617 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:53:48,619 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:53:48,619 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:53:48,619 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:48,619 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:48,621 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:53:48,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:53:48,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:53:48,621 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:48,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:48,622 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:48,622 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:53:48,622 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:48,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:53:48,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:48,623 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:53:48,624 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:53:48,624 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:53:48,624 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:53:48,624 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:53:48,629 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:53:48,629 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:53:48,629 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:53:48,629 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. 2023-04-19 20:53:48,629 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. started... 2023-04-19 20:53:48,629 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. as already flushing 2023-04-19 20:53:48,629 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. 2023-04-19 20:53:48,629 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:53:48,630 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:53:48,630 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:48,630 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:53:48,630 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:53:48,634 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:48,634 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:48,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:48,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:53:48,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:53:48,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:53:48,634 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:48,634 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:53:48,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:53:48,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:48,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:48,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:53:48,637 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:53:48,637 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@504745c1[Count = 0] remaining members to acquire global barrier 2023-04-19 20:53:48,637 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:53:48,637 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:48,638 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:48,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:48,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:48,639 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:48,639 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:53:48,639 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:53:48,639 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:53:48,639 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:53:48,642 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:53:48,642 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:53:48,642 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:53:48,647 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:48,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:48,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:53:48,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:53:48,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:53:48,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:53:48,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:48,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:48,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:53:48,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:48,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:48,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:53:48,653 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:53:48,653 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:53:48,653 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:53:48,653 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:53:48,653 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:53:48,653 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:53:48,657 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:48,657 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:53:48,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:48,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:48,657 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:48,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:53:48,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:53:48,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:48,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:53:48,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:53:48,658 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:48,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:48,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:53:48,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:48,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:53:48,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:48,661 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:48,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:48,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:53:48,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:48,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:48,666 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:48,666 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:48,666 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:48,666 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:48,666 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:53:48,666 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:53:48,666 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:53:48,666 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:48,672 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:48,672 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:48,672 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:53:48,666 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:53:48,673 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:53:48,673 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:53:48,674 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:53:48,674 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:53:48,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:53:48,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:53:48,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:53:48,774 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:48,774 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:53:48,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42342 deadline: 1681937688774, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:48,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:53:48,781 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:53:48,782 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:48,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42366 deadline: 1681937688779, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:48,783 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:48,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42350 deadline: 1681937688781, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:48,784 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:48,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42380 deadline: 1681937688782, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:48,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:53:48,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:53:48,786 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:53:48,786 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:53:48,789 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:53:48,789 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:53:48,789 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:48,789 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:48,791 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:53:48,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:53:48,791 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:48,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:53:48,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:48,792 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:48,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:53:48,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:48,793 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:53:48,793 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:48,793 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:53:48,796 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:53:48,796 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:53:48,796 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:53:48,796 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:53:48,801 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:53:48,820 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:53:48,820 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. 2023-04-19 20:53:48,820 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:53:48,820 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. started... 2023-04-19 20:53:48,820 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. as already flushing 2023-04-19 20:53:48,820 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. 2023-04-19 20:53:48,820 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:53:48,820 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:53:48,821 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:48,821 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:53:48,821 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:53:48,823 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:48,823 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:48,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:48,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:53:48,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:53:48,823 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:48,823 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:53:48,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:53:48,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:53:48,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:48,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:48,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:53:48,836 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:53:48,837 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@45b25d86[Count = 0] remaining members to acquire global barrier 2023-04-19 20:53:48,837 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:53:48,837 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:48,838 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:48,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:48,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:48,838 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:53:48,839 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:53:48,839 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:53:48,839 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:48,839 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:53:48,841 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:48,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:48,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:53:48,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:53:48,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:53:48,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:53:48,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:48,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:48,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:53:48,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:48,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:48,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:53:48,846 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:53:48,846 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:53:48,846 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:53:48,846 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:53:48,846 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:53:48,846 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:53:48,848 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:53:48,848 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:53:48,848 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:53:48,848 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:48,853 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:53:48,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:48,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:48,853 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:48,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:48,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:53:48,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:53:48,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:53:48,854 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:48,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:48,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:53:48,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:53:48,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:53:48,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:48,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:48,856 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:48,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:48,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:53:48,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:48,866 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/B/e8ee299c3faa47798edd5710a18a826f as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/e8ee299c3faa47798edd5710a18a826f 2023-04-19 20:53:48,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:48,868 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:53:48,868 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:53:48,868 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:53:48,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:53:48,868 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:53:48,868 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:53:48,871 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:53:48,868 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/A/f7ce0cda94e94edc83f1dbcf54f4e122 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/f7ce0cda94e94edc83f1dbcf54f4e122 2023-04-19 20:53:48,868 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:48,872 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:48,872 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:48,872 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:53:48,872 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:48,872 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:53:48,872 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:48,872 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:48,872 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:48,873 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:53:48,873 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:53:48,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:53:48,883 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in f70bfe945d0054ce5a228cf255d902f5/B of f70bfe945d0054ce5a228cf255d902f5 into e8ee299c3faa47798edd5710a18a826f(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 20:53:48,883 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for f70bfe945d0054ce5a228cf255d902f5: 2023-04-19 20:53:48,883 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5., storeName=f70bfe945d0054ce5a228cf255d902f5/B, priority=13, startTime=1681937628299; duration=0sec 2023-04-19 20:53:48,884 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 20:53:48,884 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-04-19 20:53:48,884 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in f70bfe945d0054ce5a228cf255d902f5/A of f70bfe945d0054ce5a228cf255d902f5 into f7ce0cda94e94edc83f1dbcf54f4e122(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 20:53:48,884 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for f70bfe945d0054ce5a228cf255d902f5: 2023-04-19 20:53:48,884 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5., storeName=f70bfe945d0054ce5a228cf255d902f5/A, priority=13, startTime=1681937628299; duration=0sec 2023-04-19 20:53:48,884 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 20:53:48,885 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-19 20:53:48,885 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-19 20:53:48,885 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. because compaction request was cancelled 2023-04-19 20:53:48,924 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=551 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/B/10a9f31a376a40e39a39033d457e01c4 2023-04-19 20:53:48,972 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:53:48,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:53:48,976 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:53:48,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:53:48,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:53:48,979 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:53:48,979 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:53:48,979 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:53:48,979 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:53:48,979 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:48,979 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:48,981 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:48,981 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:53:48,981 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:53:48,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:53:48,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:48,982 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:48,982 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:53:48,982 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:48,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:53:48,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:48,983 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:53:48,984 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:53:48,984 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:53:48,984 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:53:48,984 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:53:48,984 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:53:48,988 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:53:48,988 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:53:48,988 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. 2023-04-19 20:53:48,988 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. started... 2023-04-19 20:53:48,988 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. as already flushing 2023-04-19 20:53:48,988 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. 2023-04-19 20:53:48,988 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:53:48,988 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:53:48,988 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:48,988 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:53:48,989 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:53:48,992 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:48,992 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:48,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:48,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:53:48,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:53:48,992 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:48,992 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:53:48,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:53:48,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:53:48,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:48,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:48,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:53:48,995 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:53:48,995 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@543bb04d[Count = 0] remaining members to acquire global barrier 2023-04-19 20:53:48,995 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:53:48,995 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:48,997 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:48,997 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:48,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:48,997 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:53:48,997 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:53:48,997 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:53:48,997 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:48,997 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:53:48,999 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:48,999 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:53:48,999 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:53:49,000 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:53:48,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:49,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:53:49,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:53:49,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:53:49,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:53:49,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:49,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:49,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:53:49,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:49,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:49,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:53:49,004 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:53:49,004 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:53:49,004 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:53:49,004 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:53:49,004 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:53:49,004 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:53:49,008 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:49,008 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:49,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:49,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:49,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:53:49,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:53:49,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:49,008 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:53:49,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:53:49,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:53:49,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:53:49,009 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:49,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:49,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:49,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:53:49,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:49,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:49,011 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:49,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:53:49,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:49,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:49,018 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:53:49,018 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:53:49,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:53:49,018 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:49,018 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:53:49,018 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:53:49,018 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:49,018 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:49,018 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:49,018 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:49,018 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:49,018 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:49,018 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:53:49,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:53:49,018 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:53:49,019 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:53:49,019 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:53:49,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:53:49,020 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:53:49,079 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:49,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42342 deadline: 1681937689078, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:49,086 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:49,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42366 deadline: 1681937689085, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:49,086 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:49,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42350 deadline: 1681937689085, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:49,094 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:49,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42380 deadline: 1681937689092, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:49,119 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:53:49,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:53:49,123 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:53:49,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:53:49,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:53:49,125 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:53:49,125 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:53:49,125 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:53:49,125 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:53:49,125 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:49,125 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:49,127 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:53:49,127 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:49,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:53:49,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:53:49,127 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:49,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:49,127 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:53:49,128 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:49,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:53:49,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:49,128 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:53:49,129 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:53:49,129 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:53:49,129 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:53:49,129 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:53:49,129 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:53:49,132 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:53:49,132 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:53:49,132 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. 2023-04-19 20:53:49,132 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. started... 2023-04-19 20:53:49,132 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. as already flushing 2023-04-19 20:53:49,132 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. 2023-04-19 20:53:49,132 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:53:49,132 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:53:49,132 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:49,132 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:53:49,132 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:53:49,134 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:49,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:49,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:53:49,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:53:49,135 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:49,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:53:49,135 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:49,135 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:53:49,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:53:49,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:49,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:49,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:53:49,137 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:53:49,137 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4c4b18e6[Count = 0] remaining members to acquire global barrier 2023-04-19 20:53:49,137 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:53:49,137 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:49,139 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:49,139 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:49,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:49,139 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:53:49,139 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:53:49,139 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:53:49,139 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:49,139 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:53:49,142 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:49,142 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:53:49,142 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:53:49,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:49,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:53:49,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:53:49,142 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:53:49,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:53:49,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:53:49,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:49,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:49,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:53:49,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:49,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:49,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:53:49,145 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:53:49,145 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:53:49,145 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:53:49,145 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:53:49,145 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:53:49,145 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:53:49,147 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:49,147 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:49,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:49,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:49,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:49,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:53:49,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:53:49,147 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:53:49,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:53:49,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:53:49,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:53:49,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:49,148 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:49,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:49,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:53:49,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:49,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:49,150 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:49,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:53:49,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:49,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:49,154 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:49,154 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:49,154 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:49,154 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:49,154 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:49,154 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:49,154 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:49,154 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:53:49,154 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:53:49,154 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:53:49,154 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:53:49,154 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:53:49,155 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:53:49,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:53:49,155 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:53:49,155 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:53:49,155 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:53:49,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:53:49,255 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:53:49,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:53:49,258 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:53:49,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:53:49,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:53:49,260 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:53:49,260 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:53:49,260 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:53:49,260 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:53:49,260 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:49,260 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:49,262 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:53:49,262 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:49,262 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:53:49,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:53:49,262 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:49,262 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:53:49,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:49,263 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:49,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:53:49,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:49,264 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:53:49,264 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:53:49,264 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:53:49,264 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:53:49,265 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:53:49,265 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:53:49,265 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:53:49,265 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:53:49,265 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. 2023-04-19 20:53:49,265 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. started... 2023-04-19 20:53:49,265 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. as already flushing 2023-04-19 20:53:49,265 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. 2023-04-19 20:53:49,265 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:53:49,265 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:53:49,265 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:49,265 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:53:49,265 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:53:49,268 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:49,268 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:49,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:49,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:53:49,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:53:49,268 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:49,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:53:49,268 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:53:49,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:53:49,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:49,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:49,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:53:49,270 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:53:49,270 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@264835e3[Count = 0] remaining members to acquire global barrier 2023-04-19 20:53:49,270 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:53:49,270 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:49,271 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:49,271 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:49,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:49,271 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:53:49,271 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:53:49,271 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:49,271 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:53:49,271 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:53:49,273 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:53:49,273 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:53:49,273 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:49,273 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:53:49,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:49,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:53:49,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:53:49,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:53:49,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:53:49,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:49,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:49,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:53:49,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:49,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:49,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:53:49,276 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:53:49,276 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:53:49,276 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:53:49,276 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:53:49,276 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:53:49,276 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:53:49,278 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:49,278 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:49,278 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:49,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:49,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:53:49,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:53:49,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:49,278 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:53:49,278 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:53:49,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:53:49,278 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:49,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:53:49,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:49,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:49,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:53:49,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:49,281 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:49,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:49,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:53:49,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:49,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:49,284 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:49,284 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:49,284 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:49,284 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:49,284 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:49,284 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:49,285 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:49,285 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:53:49,285 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:53:49,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:53:49,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:53:49,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:53:49,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:53:49,287 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:53:49,287 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:53:49,287 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:53:49,287 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:53:49,287 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:53:49,288 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:53:49,359 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=551 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/C/3167eafbcb14456aa5deb5ff4d6e0fa0 2023-04-19 20:53:49,371 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/A/361fbc254d34492ea61eff57284aec32 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/361fbc254d34492ea61eff57284aec32 2023-04-19 20:53:49,382 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/361fbc254d34492ea61eff57284aec32, entries=200, sequenceid=551, filesize=14.3 K 2023-04-19 20:53:49,386 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/B/10a9f31a376a40e39a39033d457e01c4 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/10a9f31a376a40e39a39033d457e01c4 2023-04-19 20:53:49,388 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:53:49,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:53:49,393 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:53:49,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:53:49,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:53:49,399 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/10a9f31a376a40e39a39033d457e01c4, entries=150, sequenceid=551, filesize=11.9 K 2023-04-19 20:53:49,409 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:53:49,409 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:53:49,409 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:53:49,409 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:53:49,410 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/C/3167eafbcb14456aa5deb5ff4d6e0fa0 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/3167eafbcb14456aa5deb5ff4d6e0fa0 2023-04-19 20:53:49,410 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:49,410 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:49,412 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:49,413 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:53:49,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:53:49,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:53:49,413 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:49,413 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:53:49,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:49,414 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:49,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:53:49,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:49,414 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:53:49,420 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/3167eafbcb14456aa5deb5ff4d6e0fa0, entries=150, sequenceid=551, filesize=11.9 K 2023-04-19 20:53:49,420 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:53:49,420 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:53:49,420 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:53:49,420 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:53:49,420 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:53:49,421 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=107.34 KB/109920 for f70bfe945d0054ce5a228cf255d902f5 in 1078ms, sequenceid=551, compaction requested=true 2023-04-19 20:53:49,421 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for f70bfe945d0054ce5a228cf255d902f5: 2023-04-19 20:53:49,423 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 20:53:49,423 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-04-19 20:53:49,423 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-04-19 20:53:49,422 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. 2023-04-19 20:53:49,421 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:53:49,423 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:53:49,423 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. started... 2023-04-19 20:53:49,423 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing f70bfe945d0054ce5a228cf255d902f5 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-19 20:53:49,424 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f70bfe945d0054ce5a228cf255d902f5, store=A 2023-04-19 20:53:49,424 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:53:49,424 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-19 20:53:49,424 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-19 20:53:49,424 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. because compaction request was cancelled 2023-04-19 20:53:49,425 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-19 20:53:49,423 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 20:53:49,425 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-19 20:53:49,425 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-19 20:53:49,425 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. because compaction request was cancelled 2023-04-19 20:53:49,424 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f70bfe945d0054ce5a228cf255d902f5, store=B 2023-04-19 20:53:49,425 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 20:53:49,425 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:53:49,425 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f70bfe945d0054ce5a228cf255d902f5, store=C 2023-04-19 20:53:49,425 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:53:49,426 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36764 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-19 20:53:49,426 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1912): f70bfe945d0054ce5a228cf255d902f5/C is initiating minor compaction (all files) 2023-04-19 20:53:49,427 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of f70bfe945d0054ce5a228cf255d902f5/C in TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. 2023-04-19 20:53:49,427 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/492a54bd908348119b1b63b030bee115, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/5e1159c20c7a468d8890a6be2b8535b1, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/3167eafbcb14456aa5deb5ff4d6e0fa0] into tmpdir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp, totalSize=35.9 K 2023-04-19 20:53:49,427 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 492a54bd908348119b1b63b030bee115, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=515, earliestPutTs=1681937626400 2023-04-19 20:53:49,428 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 5e1159c20c7a468d8890a6be2b8535b1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=534, earliestPutTs=1681937627577 2023-04-19 20:53:49,429 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 3167eafbcb14456aa5deb5ff4d6e0fa0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=551, earliestPutTs=1681937627720 2023-04-19 20:53:49,441 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] throttle.PressureAwareThroughputController(145): f70bfe945d0054ce5a228cf255d902f5#C#compaction#30 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 20:53:49,473 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/C/b6e99014e8c34df7b74195b8cba735f0 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/b6e99014e8c34df7b74195b8cba735f0 2023-04-19 20:53:49,482 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in f70bfe945d0054ce5a228cf255d902f5/C of f70bfe945d0054ce5a228cf255d902f5 into b6e99014e8c34df7b74195b8cba735f0(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 20:53:49,483 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for f70bfe945d0054ce5a228cf255d902f5: 2023-04-19 20:53:49,483 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5., storeName=f70bfe945d0054ce5a228cf255d902f5/C, priority=13, startTime=1681937629425; duration=0sec 2023-04-19 20:53:49,483 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 20:53:49,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(9158): Flush requested on f70bfe945d0054ce5a228cf255d902f5 2023-04-19 20:53:49,588 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. as already flushing 2023-04-19 20:53:49,609 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:49,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42342 deadline: 1681937689603, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:49,610 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:49,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42366 deadline: 1681937689603, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:49,613 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:49,614 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:49,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42350 deadline: 1681937689609, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:49,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42380 deadline: 1681937689609, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:49,712 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:49,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42366 deadline: 1681937689711, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:49,713 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:49,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42342 deadline: 1681937689711, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:49,715 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:49,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42350 deadline: 1681937689715, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:49,716 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:49,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42380 deadline: 1681937689715, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:49,866 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=572 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/A/15fe198edcbf437195057b1b1ac62025 2023-04-19 20:53:49,919 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:49,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42366 deadline: 1681937689915, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:49,920 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:49,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42342 deadline: 1681937689919, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:49,920 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:49,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42380 deadline: 1681937689920, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:49,924 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:49,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42350 deadline: 1681937689922, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:49,928 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=572 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/B/1cb11e3b9ab540228aae33bb7ac4cc46 2023-04-19 20:53:49,962 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=572 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/C/575d80cfd03d415cadd11d98cea046d5 2023-04-19 20:53:49,970 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/A/15fe198edcbf437195057b1b1ac62025 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/15fe198edcbf437195057b1b1ac62025 2023-04-19 20:53:49,978 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/15fe198edcbf437195057b1b1ac62025, entries=150, sequenceid=572, filesize=11.9 K 2023-04-19 20:53:49,981 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/B/1cb11e3b9ab540228aae33bb7ac4cc46 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/1cb11e3b9ab540228aae33bb7ac4cc46 2023-04-19 20:53:49,989 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/1cb11e3b9ab540228aae33bb7ac4cc46, entries=150, sequenceid=572, filesize=11.9 K 2023-04-19 20:53:49,991 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/C/575d80cfd03d415cadd11d98cea046d5 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/575d80cfd03d415cadd11d98cea046d5 2023-04-19 20:53:49,997 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/575d80cfd03d415cadd11d98cea046d5, entries=150, sequenceid=572, filesize=11.9 K 2023-04-19 20:53:49,999 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for f70bfe945d0054ce5a228cf255d902f5 in 576ms, sequenceid=572, compaction requested=true 2023-04-19 20:53:49,999 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f70bfe945d0054ce5a228cf255d902f5: 2023-04-19 20:53:50,002 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. 2023-04-19 20:53:50,002 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:53:50,002 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:53:50,003 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:50,003 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:53:50,003 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:53:50,004 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:50,005 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:50,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:50,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:53:50,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:53:50,005 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:50,005 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:53:50,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:53:50,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:53:50,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:50,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:50,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:53:50,007 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:53:50,008 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@71c102c1[Count = 0] remaining members to acquire global barrier 2023-04-19 20:53:50,008 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:53:50,008 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:50,009 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:50,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:50,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:50,010 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:53:50,010 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:53:50,010 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:50,010 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:53:50,010 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:53:50,018 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:50,018 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:53:50,018 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:53:50,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:50,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:53:50,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:53:50,018 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:53:50,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:53:50,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:53:50,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:50,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:50,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:53:50,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:50,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:50,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:53:50,021 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:53:50,021 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:53:50,021 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:53:50,021 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:53:50,021 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:53:50,021 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:53:50,024 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:50,024 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:50,024 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:50,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:50,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:53:50,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:53:50,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:50,024 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:53:50,026 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:53:50,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:53:50,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:53:50,027 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:50,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:50,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:50,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:53:50,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:50,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:50,028 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:50,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:53:50,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:50,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:50,034 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:53:50,034 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:50,034 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:53:50,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:53:50,035 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:53:50,035 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:53:50,034 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:50,034 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:53:50,035 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:50,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:53:50,035 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:53:50,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:53:50,035 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:50,035 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:50,035 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:50,035 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:50,035 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:53:50,035 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:53:50,136 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:53:50,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:53:50,140 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:53:50,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:53:50,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:53:50,142 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:53:50,142 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:53:50,142 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:53:50,142 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:53:50,144 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:50,145 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:50,147 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:53:50,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:53:50,147 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:50,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:53:50,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:50,148 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:50,148 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:53:50,148 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:50,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:53:50,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:50,149 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:53:50,149 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:53:50,149 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:53:50,149 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:53:50,149 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:53:50,150 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:53:50,151 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:53:50,151 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:53:50,151 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. 2023-04-19 20:53:50,151 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. started... 2023-04-19 20:53:50,152 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing f70bfe945d0054ce5a228cf255d902f5 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-19 20:53:50,154 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f70bfe945d0054ce5a228cf255d902f5, store=A 2023-04-19 20:53:50,154 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:53:50,154 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f70bfe945d0054ce5a228cf255d902f5, store=B 2023-04-19 20:53:50,154 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:53:50,154 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f70bfe945d0054ce5a228cf255d902f5, store=C 2023-04-19 20:53:50,154 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:53:50,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(9158): Flush requested on f70bfe945d0054ce5a228cf255d902f5 2023-04-19 20:53:50,224 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. as already flushing 2023-04-19 20:53:50,257 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:50,258 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:50,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42380 deadline: 1681937690252, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:50,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42366 deadline: 1681937690250, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:50,258 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:50,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42350 deadline: 1681937690253, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:50,262 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:50,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42342 deadline: 1681937690258, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:50,362 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:50,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42366 deadline: 1681937690360, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:50,363 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:50,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42350 deadline: 1681937690363, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:50,364 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:50,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42380 deadline: 1681937690363, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:50,369 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:50,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42342 deadline: 1681937690364, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:50,567 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:50,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42380 deadline: 1681937690566, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:50,567 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:50,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42366 deadline: 1681937690566, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:50,570 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:50,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42350 deadline: 1681937690566, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:50,574 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:50,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42342 deadline: 1681937690572, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:50,579 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=586 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/A/5627443d993f4f9487601963a30ea217 2023-04-19 20:53:50,608 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=586 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/B/b0da787dd1ca42f89b336b3dbd2d12c5 2023-04-19 20:53:50,874 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:50,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42366 deadline: 1681937690871, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:50,875 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:50,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42380 deadline: 1681937690871, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:50,876 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:50,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42350 deadline: 1681937690874, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:50,879 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:50,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42342 deadline: 1681937690877, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:51,034 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=586 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/C/bea3bab17cb740d08624752ba4a9203c 2023-04-19 20:53:51,041 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/A/5627443d993f4f9487601963a30ea217 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/5627443d993f4f9487601963a30ea217 2023-04-19 20:53:51,047 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/5627443d993f4f9487601963a30ea217, entries=150, sequenceid=586, filesize=11.9 K 2023-04-19 20:53:51,050 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/B/b0da787dd1ca42f89b336b3dbd2d12c5 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/b0da787dd1ca42f89b336b3dbd2d12c5 2023-04-19 20:53:51,056 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/b0da787dd1ca42f89b336b3dbd2d12c5, entries=150, sequenceid=586, filesize=11.9 K 2023-04-19 20:53:51,060 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/C/bea3bab17cb740d08624752ba4a9203c as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/bea3bab17cb740d08624752ba4a9203c 2023-04-19 20:53:51,067 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/bea3bab17cb740d08624752ba4a9203c, entries=150, sequenceid=586, filesize=11.9 K 2023-04-19 20:53:51,070 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for f70bfe945d0054ce5a228cf255d902f5 in 916ms, sequenceid=586, compaction requested=true 2023-04-19 20:53:51,071 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f70bfe945d0054ce5a228cf255d902f5: 2023-04-19 20:53:51,071 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. 2023-04-19 20:53:51,071 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:53:51,075 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:53:51,075 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:51,075 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:53:51,075 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:53:51,077 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:51,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:51,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:53:51,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:53:51,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:53:51,077 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:51,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:53:51,078 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:51,078 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:53:51,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:51,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:51,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:53:51,079 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:53:51,079 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@78f17d6c[Count = 0] remaining members to acquire global barrier 2023-04-19 20:53:51,079 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:53:51,079 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:51,080 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:51,081 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:51,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:51,081 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:53:51,081 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:53:51,081 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:51,081 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:53:51,081 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:53:51,082 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:51,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:51,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:53:51,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:53:51,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:53:51,083 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:53:51,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:53:51,083 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:53:51,083 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:53:51,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:51,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:51,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:53:51,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:51,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:51,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:53:51,085 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:53:51,085 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:53:51,085 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:53:51,085 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:53:51,085 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:53:51,085 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:53:51,089 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:51,089 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:53:51,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:51,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:51,089 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:51,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:51,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:53:51,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:53:51,089 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:51,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:53:51,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:53:51,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:53:51,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:51,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:51,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:53:51,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:51,091 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:51,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:51,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:53:51,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:51,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:51,095 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:53:51,095 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:53:51,095 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:53:51,095 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:53:51,095 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:53:51,095 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:53:51,095 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:51,096 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:51,096 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:51,096 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:51,096 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:51,096 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:51,096 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:51,096 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:53:51,096 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:53:51,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:53:51,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:53:51,097 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:53:51,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:53:51,195 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:53:51,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:53:51,198 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:53:51,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:53:51,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:53:51,199 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:53:51,199 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:53:51,199 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:53:51,199 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:53:51,200 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:51,200 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:51,201 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:53:51,201 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:51,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:53:51,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:53:51,201 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:51,201 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:53:51,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:51,202 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:51,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:53:51,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:51,202 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:53:51,203 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:53:51,203 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:53:51,203 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:53:51,203 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:53:51,203 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:53:51,203 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:53:51,203 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:53:51,203 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. 2023-04-19 20:53:51,203 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. started... 2023-04-19 20:53:51,203 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing f70bfe945d0054ce5a228cf255d902f5 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-19 20:53:51,204 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f70bfe945d0054ce5a228cf255d902f5, store=A 2023-04-19 20:53:51,204 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:53:51,204 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f70bfe945d0054ce5a228cf255d902f5, store=B 2023-04-19 20:53:51,204 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:53:51,204 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f70bfe945d0054ce5a228cf255d902f5, store=C 2023-04-19 20:53:51,204 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:53:51,237 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=605 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/A/c7899702bdf34f54862407fd3c7dcc91 2023-04-19 20:53:51,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(9158): Flush requested on f70bfe945d0054ce5a228cf255d902f5 2023-04-19 20:53:51,384 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. as already flushing 2023-04-19 20:53:51,397 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:51,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42342 deadline: 1681937691393, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:51,398 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:51,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42380 deadline: 1681937691393, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:51,400 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:51,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42350 deadline: 1681937691396, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:51,401 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:51,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42366 deadline: 1681937691397, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:51,502 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:51,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42342 deadline: 1681937691499, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:51,503 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:51,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42380 deadline: 1681937691499, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:51,504 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:51,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42350 deadline: 1681937691502, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:51,505 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:51,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42366 deadline: 1681937691503, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:51,637 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:51,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42384 deadline: 1681937691636, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:51,639 DEBUG [Thread-141] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8263 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5., hostname=jenkins-hbase4.apache.org,35247,1681937616971, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-19 20:53:51,686 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=605 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/B/6d00dd51967e4100b40f46aec06d43b7 2023-04-19 20:53:51,704 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:51,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42342 deadline: 1681937691704, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:51,710 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:51,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42366 deadline: 1681937691709, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:51,711 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:51,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42380 deadline: 1681937691710, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:51,717 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:51,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42350 deadline: 1681937691711, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:52,015 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:52,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42342 deadline: 1681937692013, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:52,020 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:52,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42366 deadline: 1681937692018, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:52,020 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:52,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42380 deadline: 1681937692018, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:52,021 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:52,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42350 deadline: 1681937692021, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:52,121 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=605 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/C/69f71eb7bce9460fa12e370dce2a1ccb 2023-04-19 20:53:52,131 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/A/c7899702bdf34f54862407fd3c7dcc91 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/c7899702bdf34f54862407fd3c7dcc91 2023-04-19 20:53:52,138 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/c7899702bdf34f54862407fd3c7dcc91, entries=150, sequenceid=605, filesize=11.9 K 2023-04-19 20:53:52,144 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/B/6d00dd51967e4100b40f46aec06d43b7 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/6d00dd51967e4100b40f46aec06d43b7 2023-04-19 20:53:52,153 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/6d00dd51967e4100b40f46aec06d43b7, entries=150, sequenceid=605, filesize=11.9 K 2023-04-19 20:53:52,154 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/C/69f71eb7bce9460fa12e370dce2a1ccb as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/69f71eb7bce9460fa12e370dce2a1ccb 2023-04-19 20:53:52,160 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/69f71eb7bce9460fa12e370dce2a1ccb, entries=150, sequenceid=605, filesize=11.9 K 2023-04-19 20:53:52,161 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=73.80 KB/75570 for f70bfe945d0054ce5a228cf255d902f5 in 958ms, sequenceid=605, compaction requested=true 2023-04-19 20:53:52,161 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f70bfe945d0054ce5a228cf255d902f5: 2023-04-19 20:53:52,162 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. 2023-04-19 20:53:52,162 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:53:52,165 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:53:52,165 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:52,165 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:53:52,165 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:53:52,167 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:52,167 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:52,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:52,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:53:52,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:53:52,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:53:52,168 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:52,168 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:53:52,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:53:52,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:52,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:52,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:53:52,169 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:53:52,169 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6c69d6bc[Count = 0] remaining members to acquire global barrier 2023-04-19 20:53:52,169 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:53:52,170 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:52,171 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:52,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:52,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:52,171 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:53:52,171 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:53:52,171 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:53:52,171 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:52,171 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:53:52,173 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:53:52,173 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:53:52,173 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:53:52,173 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:52,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:52,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:53:52,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:53:52,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:53:52,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:53:52,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:52,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:52,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:53:52,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:52,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:52,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:53:52,177 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:53:52,178 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:53:52,178 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:53:52,178 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:53:52,178 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:53:52,178 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:53:52,181 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:52,181 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:53:52,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:52,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:52,181 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:52,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:52,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:53:52,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:53:52,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:53:52,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:53:52,182 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:52,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:53:52,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:52,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:52,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:53:52,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:52,184 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:52,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:52,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:53:52,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:52,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:52,186 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:53:52,186 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:53:52,186 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:53:52,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:53:52,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:53:52,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:53:52,187 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:52,187 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:53:52,187 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:53:52,187 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:53:52,187 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:52,188 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:52,188 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:52,188 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:52,188 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:52,188 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:53:52,188 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:53:52,188 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:52,288 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:53:52,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:53:52,301 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:53:52,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:53:52,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:53:52,316 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:53:52,316 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:53:52,316 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:53:52,316 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:53:52,317 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:52,317 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:52,319 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:52,319 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:53:52,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:53:52,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:53:52,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:52,320 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:52,320 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:53:52,320 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:52,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:53:52,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:52,321 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:53:52,332 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:53:52,332 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:53:52,332 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:53:52,332 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:53:52,332 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:53:52,332 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:53:52,332 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. 2023-04-19 20:53:52,333 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. started... 2023-04-19 20:53:52,332 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:53:52,333 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing f70bfe945d0054ce5a228cf255d902f5 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-19 20:53:52,333 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f70bfe945d0054ce5a228cf255d902f5, store=A 2023-04-19 20:53:52,333 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:53:52,333 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f70bfe945d0054ce5a228cf255d902f5, store=B 2023-04-19 20:53:52,333 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:53:52,334 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f70bfe945d0054ce5a228cf255d902f5, store=C 2023-04-19 20:53:52,334 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:53:52,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(9158): Flush requested on f70bfe945d0054ce5a228cf255d902f5 2023-04-19 20:53:52,519 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. as already flushing 2023-04-19 20:53:52,544 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:52,544 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:52,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42350 deadline: 1681937692539, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:52,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42342 deadline: 1681937692540, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:52,549 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:52,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42366 deadline: 1681937692544, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:52,549 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:52,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42380 deadline: 1681937692544, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:52,648 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:52,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42342 deadline: 1681937692645, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:52,648 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:52,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42350 deadline: 1681937692645, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:52,651 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:52,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42366 deadline: 1681937692650, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:52,654 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:52,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42380 deadline: 1681937692651, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:52,771 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=619 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/A/5860593eba5b479fb5f432352eca0a37 2023-04-19 20:53:52,853 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:52,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42342 deadline: 1681937692849, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:52,856 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:52,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42350 deadline: 1681937692850, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:52,857 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:52,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42366 deadline: 1681937692852, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:52,857 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:52,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42380 deadline: 1681937692856, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:53,156 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:53,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42342 deadline: 1681937693155, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:53,159 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:53,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42350 deadline: 1681937693159, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:53,160 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:53,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42366 deadline: 1681937693160, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:53,163 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:53,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42380 deadline: 1681937693161, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:53,187 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=619 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/B/8a009e695e884b3c849580f60c78f004 2023-04-19 20:53:53,210 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=619 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/C/6f0002a2b36d44f2aff335d297fbdae3 2023-04-19 20:53:53,219 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/A/5860593eba5b479fb5f432352eca0a37 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/5860593eba5b479fb5f432352eca0a37 2023-04-19 20:53:53,227 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/5860593eba5b479fb5f432352eca0a37, entries=150, sequenceid=619, filesize=11.9 K 2023-04-19 20:53:53,229 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/B/8a009e695e884b3c849580f60c78f004 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/8a009e695e884b3c849580f60c78f004 2023-04-19 20:53:53,248 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/8a009e695e884b3c849580f60c78f004, entries=150, sequenceid=619, filesize=11.9 K 2023-04-19 20:53:53,250 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/C/6f0002a2b36d44f2aff335d297fbdae3 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/6f0002a2b36d44f2aff335d297fbdae3 2023-04-19 20:53:53,256 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/6f0002a2b36d44f2aff335d297fbdae3, entries=150, sequenceid=619, filesize=11.9 K 2023-04-19 20:53:53,262 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for f70bfe945d0054ce5a228cf255d902f5 in 929ms, sequenceid=619, compaction requested=true 2023-04-19 20:53:53,263 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f70bfe945d0054ce5a228cf255d902f5: 2023-04-19 20:53:53,263 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. 2023-04-19 20:53:53,263 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:53:53,273 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:53:53,273 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:53,273 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:53:53,273 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:53:53,275 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:53,275 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:53,276 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:53,276 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:53:53,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:53,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:53:53,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:53:53,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:53:53,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:53:53,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:53,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:53,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:53:53,284 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:53:53,284 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@be911ff[Count = 0] remaining members to acquire global barrier 2023-04-19 20:53:53,284 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:53:53,284 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:53,286 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:53,286 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:53,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:53,286 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:53:53,287 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:53:53,287 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:53:53,287 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:53,287 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:53:53,288 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:53,288 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:53:53,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:53,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:53:53,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:53:53,288 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:53:53,289 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:53:53,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:53:53,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:53:53,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:53,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:53,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:53:53,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:53,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:53,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:53:53,292 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:53:53,292 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:53:53,292 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:53:53,292 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:53:53,293 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:53:53,293 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:53:53,295 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:53,295 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:53,295 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:53:53,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:53,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:53,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:53,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:53:53,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:53:53,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:53:53,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:53:53,296 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:53,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:53:53,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:53,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:53,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:53:53,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:53,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:53,299 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:53,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:53:53,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:53,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:53,302 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:53:53,302 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:53:53,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:53:53,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:53:53,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:53:53,304 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:53:53,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:53:53,304 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:53:53,304 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:53:53,304 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:53:53,304 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:53:53,304 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:53:53,306 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:53,306 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:53,306 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:53,306 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:53,306 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:53,306 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:53,306 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:53,404 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:53:53,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:53:53,408 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:53:53,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:53:53,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:53:53,414 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:53:53,414 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:53:53,414 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:53:53,414 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:53:53,415 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:53,415 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:53,417 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:53:53,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:53:53,417 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:53,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:53:53,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:53,418 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:53,418 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:53:53,418 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:53,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:53:53,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:53,419 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:53:53,419 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:53:53,419 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:53:53,419 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:53:53,419 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:53:53,419 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:53:53,420 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:53:53,420 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:53:53,420 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. 2023-04-19 20:53:53,420 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. started... 2023-04-19 20:53:53,420 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing f70bfe945d0054ce5a228cf255d902f5 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-19 20:53:53,420 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f70bfe945d0054ce5a228cf255d902f5, store=A 2023-04-19 20:53:53,420 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:53:53,421 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f70bfe945d0054ce5a228cf255d902f5, store=B 2023-04-19 20:53:53,421 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:53:53,421 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f70bfe945d0054ce5a228cf255d902f5, store=C 2023-04-19 20:53:53,421 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:53:53,441 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=637 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/A/dc9ee4d0171144e690af5026300de9cd 2023-04-19 20:53:53,497 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=637 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/B/aef57242c1d54d128a4795354e0dc257 2023-04-19 20:53:53,577 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=637 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/C/3f3b84850e0344d0aff253a68946da07 2023-04-19 20:53:53,586 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/A/dc9ee4d0171144e690af5026300de9cd as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/dc9ee4d0171144e690af5026300de9cd 2023-04-19 20:53:53,594 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/dc9ee4d0171144e690af5026300de9cd, entries=150, sequenceid=637, filesize=11.9 K 2023-04-19 20:53:53,595 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/B/aef57242c1d54d128a4795354e0dc257 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/aef57242c1d54d128a4795354e0dc257 2023-04-19 20:53:53,603 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/aef57242c1d54d128a4795354e0dc257, entries=150, sequenceid=637, filesize=11.9 K 2023-04-19 20:53:53,609 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/C/3f3b84850e0344d0aff253a68946da07 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/3f3b84850e0344d0aff253a68946da07 2023-04-19 20:53:53,626 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/3f3b84850e0344d0aff253a68946da07, entries=150, sequenceid=637, filesize=11.9 K 2023-04-19 20:53:53,627 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=0 B/0 for f70bfe945d0054ce5a228cf255d902f5 in 207ms, sequenceid=637, compaction requested=true 2023-04-19 20:53:53,627 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f70bfe945d0054ce5a228cf255d902f5: 2023-04-19 20:53:53,627 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. 2023-04-19 20:53:53,627 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:53:53,633 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:53:53,633 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:53,633 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:53:53,633 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:53:53,642 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:53,642 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:53,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:53,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:53:53,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:53:53,643 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:53,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:53:53,643 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:53:53,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:53:53,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:53,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:53,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:53:53,645 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:53:53,645 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@72c7cd49[Count = 0] remaining members to acquire global barrier 2023-04-19 20:53:53,645 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:53:53,645 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:53,647 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:53,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:53,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:53,647 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:53:53,647 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:53:53,647 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:53:53,647 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:53,647 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:53:53,652 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:53:53,652 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:53:53,652 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:53:53,652 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:53,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:53,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:53:53,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:53:53,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:53:53,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:53:53,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:53,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:53,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:53:53,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:53,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:53,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:53:53,658 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:53:53,658 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:53:53,658 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:53:53,658 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:53:53,658 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:53:53,658 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:53:53,662 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:53,662 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:53,662 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:53,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:53,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:53,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:53:53,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:53:53,662 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:53:53,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:53:53,663 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:53:53,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:53:53,663 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:53,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:53,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:53,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:53:53,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:53,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:53,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:53:53,666 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:53,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:53,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:53,670 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:53:53,670 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:53:53,670 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:53:53,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:53:53,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:53:53,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:53:53,672 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:53,672 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:53,672 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:53,672 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:53:53,672 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:53:53,672 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:53:53,672 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:53,672 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:53,672 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:53,672 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:53,672 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:53:53,672 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:53:53,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(9158): Flush requested on f70bfe945d0054ce5a228cf255d902f5 2023-04-19 20:53:53,684 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing f70bfe945d0054ce5a228cf255d902f5 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-19 20:53:53,688 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f70bfe945d0054ce5a228cf255d902f5, store=A 2023-04-19 20:53:53,688 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:53:53,688 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f70bfe945d0054ce5a228cf255d902f5, store=B 2023-04-19 20:53:53,688 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:53:53,688 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f70bfe945d0054ce5a228cf255d902f5, store=C 2023-04-19 20:53:53,688 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:53:53,741 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=648 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/A/9266d1eeb42b440eaa5084e68480614a 2023-04-19 20:53:53,747 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14144, totalHeapSize=32376, min timestamp=1681937633737, max timestamp=1681937633745Num uniques 859; 2023-04-19 20:53:53,748 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=14144, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-19 20:53:53,751 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1681937633737, max timestamp=1681937633745Num uniques 859; 2023-04-19 20:53:53,751 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-19 20:53:53,752 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1681937633737, max timestamp=1681937633745Num uniques 859; 2023-04-19 20:53:53,752 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-19 20:53:53,768 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=648 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/B/7f29115fa18c49d18d69a6ea9ac024a7 2023-04-19 20:53:53,773 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:53:53,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:53:53,777 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:53:53,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:53:53,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:53:53,779 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:53:53,779 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:53:53,779 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:53:53,779 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:53:53,779 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:53,779 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:53,781 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:53,781 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:53:53,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:53:53,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:53:53,781 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:53,781 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:53:53,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:53,782 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:53,782 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:53,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42350 deadline: 1681937693774, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:53,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:53:53,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:53,783 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:53:53,784 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:53:53,784 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:53:53,784 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:53:53,784 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:53:53,784 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:53:53,787 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:53,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42380 deadline: 1681937693777, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:53,787 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:53,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42366 deadline: 1681937693779, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:53,788 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:53:53,788 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:53:53,788 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. 2023-04-19 20:53:53,788 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. started... 2023-04-19 20:53:53,788 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. as already flushing 2023-04-19 20:53:53,788 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. 2023-04-19 20:53:53,788 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:53:53,788 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:53:53,788 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:53,788 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:53:53,788 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:53:53,789 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:53,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42342 deadline: 1681937693782, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:53,792 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:53,793 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:53,793 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:53,793 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:53:53,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:53,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:53:53,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:53:53,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:53:53,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:53:53,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:53,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:53,796 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=648 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/C/9a206002c16642d3bd46da50c12a4c8d 2023-04-19 20:53:53,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:53:53,799 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:53:53,799 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@12ed460[Count = 0] remaining members to acquire global barrier 2023-04-19 20:53:53,799 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:53:53,799 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:53,800 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:53,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:53,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:53,801 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:53,801 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:53:53,801 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:53:53,801 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:53:53,801 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:53:53,803 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:53:53,803 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:53:53,803 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:53:53,803 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:53,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:53,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:53:53,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:53:53,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:53:53,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:53:53,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:53,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:53,805 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/A/9266d1eeb42b440eaa5084e68480614a as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/9266d1eeb42b440eaa5084e68480614a 2023-04-19 20:53:53,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:53:53,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:53,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:53,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:53:53,806 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:53:53,806 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:53:53,806 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:53:53,806 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:53:53,806 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:53:53,806 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:53:53,808 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:53,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:53,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:53:53,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:53:53,809 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:53,809 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:53:53,809 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:53,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:53,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:53:53,809 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:53,809 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:53:53,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:53:53,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:53,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:53,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:53:53,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:53,811 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:53,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:53,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:53:53,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:53,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:53,813 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/9266d1eeb42b440eaa5084e68480614a, entries=250, sequenceid=648, filesize=16.7 K 2023-04-19 20:53:53,814 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/B/7f29115fa18c49d18d69a6ea9ac024a7 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/7f29115fa18c49d18d69a6ea9ac024a7 2023-04-19 20:53:53,815 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:53:53,815 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:53,815 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:53,815 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:53,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:53:53,815 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:53:53,815 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:53,816 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:53,816 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:53:53,815 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:53:53,816 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:53:53,815 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:53:53,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:53:53,816 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:53,816 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:53:53,816 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:53:53,816 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:53,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:53:53,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:53:53,822 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/7f29115fa18c49d18d69a6ea9ac024a7, entries=150, sequenceid=648, filesize=11.9 K 2023-04-19 20:53:53,823 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/C/9a206002c16642d3bd46da50c12a4c8d as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/9a206002c16642d3bd46da50c12a4c8d 2023-04-19 20:53:53,834 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/9a206002c16642d3bd46da50c12a4c8d, entries=150, sequenceid=648, filesize=11.9 K 2023-04-19 20:53:53,836 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=134.18 KB/137400 for f70bfe945d0054ce5a228cf255d902f5 in 152ms, sequenceid=648, compaction requested=true 2023-04-19 20:53:53,836 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for f70bfe945d0054ce5a228cf255d902f5: 2023-04-19 20:53:53,836 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 20:53:53,836 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 20:53:53,836 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 8 store files, 0 compacting, 8 eligible, 16 blocking 2023-04-19 20:53:53,836 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-19 20:53:53,836 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 8 store files, 0 compacting, 8 eligible, 16 blocking 2023-04-19 20:53:53,839 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 8 files of size 105082 starting at candidate #0 after considering 21 permutations with 21 in ratio 2023-04-19 20:53:53,839 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1912): f70bfe945d0054ce5a228cf255d902f5/A is initiating minor compaction (all files) 2023-04-19 20:53:53,839 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 8 files of size 97762 starting at candidate #0 after considering 21 permutations with 21 in ratio 2023-04-19 20:53:53,839 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of f70bfe945d0054ce5a228cf255d902f5/A in TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. 2023-04-19 20:53:53,840 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1912): f70bfe945d0054ce5a228cf255d902f5/B is initiating minor compaction (all files) 2023-04-19 20:53:53,840 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2259): Starting compaction of f70bfe945d0054ce5a228cf255d902f5/B in TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. 2023-04-19 20:53:53,840 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/f7ce0cda94e94edc83f1dbcf54f4e122, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/361fbc254d34492ea61eff57284aec32, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/15fe198edcbf437195057b1b1ac62025, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/5627443d993f4f9487601963a30ea217, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/c7899702bdf34f54862407fd3c7dcc91, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/5860593eba5b479fb5f432352eca0a37, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/dc9ee4d0171144e690af5026300de9cd, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/9266d1eeb42b440eaa5084e68480614a] into tmpdir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp, totalSize=102.6 K 2023-04-19 20:53:53,840 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/e8ee299c3faa47798edd5710a18a826f, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/10a9f31a376a40e39a39033d457e01c4, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/1cb11e3b9ab540228aae33bb7ac4cc46, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/b0da787dd1ca42f89b336b3dbd2d12c5, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/6d00dd51967e4100b40f46aec06d43b7, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/8a009e695e884b3c849580f60c78f004, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/aef57242c1d54d128a4795354e0dc257, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/7f29115fa18c49d18d69a6ea9ac024a7] into tmpdir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp, totalSize=95.5 K 2023-04-19 20:53:53,840 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting f7ce0cda94e94edc83f1dbcf54f4e122, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=534, earliestPutTs=1681937627577 2023-04-19 20:53:53,843 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 361fbc254d34492ea61eff57284aec32, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=551, earliestPutTs=1681937627713 2023-04-19 20:53:53,843 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 15fe198edcbf437195057b1b1ac62025, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=572, earliestPutTs=1681937628368 2023-04-19 20:53:53,844 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting e8ee299c3faa47798edd5710a18a826f, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=534, earliestPutTs=1681937627577 2023-04-19 20:53:53,850 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 5627443d993f4f9487601963a30ea217, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=586, earliestPutTs=1681937629591 2023-04-19 20:53:53,850 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 10a9f31a376a40e39a39033d457e01c4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=551, earliestPutTs=1681937627720 2023-04-19 20:53:53,851 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 1cb11e3b9ab540228aae33bb7ac4cc46, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=572, earliestPutTs=1681937628368 2023-04-19 20:53:53,851 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting b0da787dd1ca42f89b336b3dbd2d12c5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=586, earliestPutTs=1681937629591 2023-04-19 20:53:53,852 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting c7899702bdf34f54862407fd3c7dcc91, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=605, earliestPutTs=1681937630241 2023-04-19 20:53:53,852 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 6d00dd51967e4100b40f46aec06d43b7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=605, earliestPutTs=1681937630241 2023-04-19 20:53:53,852 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 5860593eba5b479fb5f432352eca0a37, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=619, earliestPutTs=1681937631391 2023-04-19 20:53:53,853 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 8a009e695e884b3c849580f60c78f004, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=619, earliestPutTs=1681937631391 2023-04-19 20:53:53,853 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting dc9ee4d0171144e690af5026300de9cd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=637, earliestPutTs=1681937632539 2023-04-19 20:53:53,853 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting aef57242c1d54d128a4795354e0dc257, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=637, earliestPutTs=1681937632539 2023-04-19 20:53:53,853 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 9266d1eeb42b440eaa5084e68480614a, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=648, earliestPutTs=1681937633674 2023-04-19 20:53:53,854 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 7f29115fa18c49d18d69a6ea9ac024a7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=648, earliestPutTs=1681937633680 2023-04-19 20:53:53,880 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] throttle.PressureAwareThroughputController(145): f70bfe945d0054ce5a228cf255d902f5#A#compaction#48 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 20:53:53,888 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] throttle.PressureAwareThroughputController(145): f70bfe945d0054ce5a228cf255d902f5#B#compaction#49 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 20:53:53,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(9158): Flush requested on f70bfe945d0054ce5a228cf255d902f5 2023-04-19 20:53:53,890 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing f70bfe945d0054ce5a228cf255d902f5 3/3 column families, dataSize=140.89 KB heapSize=390.84 KB 2023-04-19 20:53:53,890 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f70bfe945d0054ce5a228cf255d902f5, store=A 2023-04-19 20:53:53,891 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 20:53:53,891 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f70bfe945d0054ce5a228cf255d902f5, store=B 2023-04-19 20:53:53,891 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 20:53:53,891 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f70bfe945d0054ce5a228cf255d902f5, store=C 2023-04-19 20:53:53,891 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 20:53:53,915 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:53,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42366 deadline: 1681937693902, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:53,915 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:53,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42380 deadline: 1681937693904, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:53,916 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:53,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42350 deadline: 1681937693909, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:53,916 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:53:53,917 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:53,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42342 deadline: 1681937693910, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:53,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:53:53,921 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:53:53,953 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/A/78780e4b985840c18ce9ab9115c522dd as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/78780e4b985840c18ce9ab9115c522dd 2023-04-19 20:53:53,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:53:53,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:53:53,968 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:53:53,968 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:53:53,968 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:53:53,968 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:53:53,969 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:53,969 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:53,971 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:53:53,971 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:53:53,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:53:53,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:53,972 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 8 (all) file(s) in f70bfe945d0054ce5a228cf255d902f5/A of f70bfe945d0054ce5a228cf255d902f5 into 78780e4b985840c18ce9ab9115c522dd(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 20:53:53,972 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for f70bfe945d0054ce5a228cf255d902f5: 2023-04-19 20:53:53,972 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5., storeName=f70bfe945d0054ce5a228cf255d902f5/A, priority=8, startTime=1681937633836; duration=0sec 2023-04-19 20:53:53,972 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 20:53:53,972 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-19 20:53:53,974 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:53,974 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:53,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:53:53,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:53,975 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:53:53,975 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:53,975 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:53:53,976 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 85610 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-19 20:53:53,977 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1912): f70bfe945d0054ce5a228cf255d902f5/C is initiating minor compaction (all files) 2023-04-19 20:53:53,977 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of f70bfe945d0054ce5a228cf255d902f5/C in TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. 2023-04-19 20:53:53,977 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/b6e99014e8c34df7b74195b8cba735f0, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/575d80cfd03d415cadd11d98cea046d5, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/bea3bab17cb740d08624752ba4a9203c, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/69f71eb7bce9460fa12e370dce2a1ccb, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/6f0002a2b36d44f2aff335d297fbdae3, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/3f3b84850e0344d0aff253a68946da07, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/9a206002c16642d3bd46da50c12a4c8d] into tmpdir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp, totalSize=83.6 K 2023-04-19 20:53:53,977 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting b6e99014e8c34df7b74195b8cba735f0, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=551, earliestPutTs=1681937627720 2023-04-19 20:53:53,978 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 575d80cfd03d415cadd11d98cea046d5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=572, earliestPutTs=1681937628368 2023-04-19 20:53:53,979 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting bea3bab17cb740d08624752ba4a9203c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=586, earliestPutTs=1681937629591 2023-04-19 20:53:53,975 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:53:53,981 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:53:53,981 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:53:53,981 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:53:53,982 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:53:53,982 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:53:53,982 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:53:53,982 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. 2023-04-19 20:53:53,982 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. started... 2023-04-19 20:53:53,982 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. as already flushing 2023-04-19 20:53:53,982 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. 2023-04-19 20:53:53,982 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 69f71eb7bce9460fa12e370dce2a1ccb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=605, earliestPutTs=1681937630241 2023-04-19 20:53:53,982 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:53:53,983 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:53:53,983 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:53,983 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:53:53,983 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:53:53,983 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 6f0002a2b36d44f2aff335d297fbdae3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=619, earliestPutTs=1681937631391 2023-04-19 20:53:53,984 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 3f3b84850e0344d0aff253a68946da07, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=637, earliestPutTs=1681937632539 2023-04-19 20:53:53,984 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 9a206002c16642d3bd46da50c12a4c8d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=648, earliestPutTs=1681937633680 2023-04-19 20:53:53,995 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:53,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:53,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:53:53,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:53:53,996 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:53,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:53:53,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:53:53,996 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:53,996 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:53:53,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:53,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:53,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:53:53,999 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:53:53,999 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@23d03ecf[Count = 0] remaining members to acquire global barrier 2023-04-19 20:53:53,999 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:53:53,999 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:54,001 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:54,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:54,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:54,001 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:53:54,001 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:53:54,001 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:53:54,002 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:54,002 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:53:54,006 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] throttle.PressureAwareThroughputController(145): f70bfe945d0054ce5a228cf255d902f5#C#compaction#51 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 20:53:54,006 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:53:54,006 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:53:54,006 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:53:54,006 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:54,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:54,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:53:54,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:53:54,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:53:54,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:53:54,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:54,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:54,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:53:54,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:54,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:54,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:53:54,011 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:53:54,011 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:53:54,012 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:53:54,012 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:53:54,012 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:53:54,012 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:53:54,016 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:54,016 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:53:54,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:54,016 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:54,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:54,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:54,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:53:54,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:53:54,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:53:54,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:53:54,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:54,018 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:54,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:53:54,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:54,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:53:54,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:54,019 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:54,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:54,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:53:54,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:54,021 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:54,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42366 deadline: 1681937694017, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:54,021 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:54,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:54,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42380 deadline: 1681937694017, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:54,022 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:54,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42350 deadline: 1681937694018, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:54,022 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:54,022 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:54,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 233 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42342 deadline: 1681937694018, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:54,022 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:54,022 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:54,022 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:54,022 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:54,022 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:54,022 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:54,022 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:53:54,022 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:53:54,022 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:53:54,023 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:53:54,022 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:53:54,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:53:54,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:53:54,023 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:53:54,023 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:53:54,023 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:53:54,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:53:54,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:53:54,123 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:53:54,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:53:54,126 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:53:54,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:53:54,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:53:54,132 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:53:54,132 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:53:54,132 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:53:54,132 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:53:54,133 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:54,133 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:54,135 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:53:54,135 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:54,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:53:54,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:53:54,135 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:54,135 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:53:54,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:54,136 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:54,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:53:54,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:54,140 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:53:54,141 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:53:54,141 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:53:54,141 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:53:54,141 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:53:54,141 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:53:54,141 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:53:54,141 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:53:54,141 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. 2023-04-19 20:53:54,141 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. started... 2023-04-19 20:53:54,141 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. as already flushing 2023-04-19 20:53:54,141 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. 2023-04-19 20:53:54,141 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:53:54,141 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:53:54,141 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:54,142 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:53:54,142 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:53:54,144 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:54,144 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:54,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:54,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:53:54,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:53:54,145 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:54,145 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:53:54,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:53:54,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:53:54,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:54,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:54,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:53:54,149 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:53:54,149 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@60f5b5a3[Count = 0] remaining members to acquire global barrier 2023-04-19 20:53:54,149 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:53:54,149 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:54,153 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:54,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:54,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:54,153 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:53:54,153 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:53:54,153 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:53:54,153 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:54,153 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:53:54,155 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:54,155 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:53:54,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:54,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:53:54,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:53:54,155 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:53:54,155 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:53:54,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:53:54,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:53:54,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:54,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:54,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:53:54,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:54,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:54,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:53:54,161 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:53:54,161 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:53:54,161 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:53:54,162 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:53:54,162 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:53:54,162 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:53:54,165 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:54,165 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:53:54,165 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:54,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:54,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:54,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:53:54,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:53:54,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:54,166 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:54,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:53:54,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:53:54,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:54,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:53:54,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:54,168 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:54,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:53:54,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:54,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:54,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:53:54,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:54,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:54,173 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:53:54,173 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:53:54,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:53:54,174 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:54,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:53:54,174 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:53:54,174 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:54,174 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:53:54,174 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:54,174 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:53:54,174 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:54,174 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:54,174 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:54,174 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:54,174 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:53:54,174 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:53:54,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:53:54,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:53:54,224 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:54,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42366 deadline: 1681937694223, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:54,224 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:54,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42350 deadline: 1681937694224, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:54,227 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:54,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 235 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42342 deadline: 1681937694225, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:54,231 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:54,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42380 deadline: 1681937694228, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:54,274 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:53:54,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:53:54,280 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:53:54,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:53:54,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:53:54,291 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:53:54,291 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:53:54,291 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:53:54,291 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:53:54,292 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:54,292 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:54,294 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:53:54,294 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:54,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:53:54,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:53:54,294 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:54,294 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:53:54,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:54,295 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:54,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:53:54,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:54,295 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:53:54,299 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:53:54,299 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:53:54,299 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:53:54,299 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:53:54,299 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:53:54,300 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:53:54,300 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:53:54,301 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. 2023-04-19 20:53:54,301 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. started... 2023-04-19 20:53:54,301 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. as already flushing 2023-04-19 20:53:54,301 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. 2023-04-19 20:53:54,301 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:53:54,302 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:53:54,302 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:54,302 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:53:54,302 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:53:54,304 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:54,304 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:54,304 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:53:54,305 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:54,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:54,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:53:54,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:53:54,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:53:54,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:53:54,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:54,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:54,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:53:54,308 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:53:54,308 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2c949171[Count = 0] remaining members to acquire global barrier 2023-04-19 20:53:54,309 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:53:54,309 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:54,311 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:54,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:54,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:54,311 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:53:54,311 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:53:54,311 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:54,312 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:53:54,311 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:53:54,314 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:54,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:54,314 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:53:54,314 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:53:54,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:53:54,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:53:54,314 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:53:54,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:53:54,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:53:54,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:54,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:54,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:53:54,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:54,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:54,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:53:54,319 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:53:54,319 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:53:54,319 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:53:54,319 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:53:54,319 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:53:54,319 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:53:54,328 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:54,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:54,328 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:54,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:53:54,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:53:54,329 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:54,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:54,328 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:53:54,329 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:53:54,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:53:54,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:53:54,330 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:54,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:54,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:54,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:53:54,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:54,332 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:54,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:54,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:53:54,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:54,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:54,337 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/B/e7b5f69e860f4c1aa8d50776099e1538 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/e7b5f69e860f4c1aa8d50776099e1538 2023-04-19 20:53:54,337 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:54,338 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:53:54,338 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:53:54,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:53:54,338 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:53:54,338 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:53:54,338 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:53:54,338 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:54,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:53:54,338 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:54,338 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:54,338 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:54,338 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:54,338 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:54,338 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:53:54,338 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:53:54,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:53:54,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:53:54,345 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1652): Completed compaction of 8 (all) file(s) in f70bfe945d0054ce5a228cf255d902f5/B of f70bfe945d0054ce5a228cf255d902f5 into e7b5f69e860f4c1aa8d50776099e1538(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 20:53:54,345 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for f70bfe945d0054ce5a228cf255d902f5: 2023-04-19 20:53:54,345 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5., storeName=f70bfe945d0054ce5a228cf255d902f5/B, priority=8, startTime=1681937633836; duration=0sec 2023-04-19 20:53:54,345 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 20:53:54,379 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=683 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/A/4d6f3b70db424f4a84619f347aedb87b 2023-04-19 20:53:54,412 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=683 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/B/b7a46c57b6be4fc1bbf2ec6393740cbb 2023-04-19 20:53:54,433 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/C/fc84d90f0d544b1dab4c05b7edfd112f as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/fc84d90f0d544b1dab4c05b7edfd112f 2023-04-19 20:53:54,438 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:53:54,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:53:54,441 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in f70bfe945d0054ce5a228cf255d902f5/C of f70bfe945d0054ce5a228cf255d902f5 into fc84d90f0d544b1dab4c05b7edfd112f(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 20:53:54,441 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for f70bfe945d0054ce5a228cf255d902f5: 2023-04-19 20:53:54,441 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5., storeName=f70bfe945d0054ce5a228cf255d902f5/C, priority=9, startTime=1681937633836; duration=0sec 2023-04-19 20:53:54,442 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 20:53:54,443 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:53:54,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:53:54,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:53:54,468 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:53:54,468 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:53:54,468 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:53:54,468 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:53:54,469 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=683 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/C/34214c7d26fa4e4082a99c85a1f29782 2023-04-19 20:53:54,470 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:54,470 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:54,472 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:53:54,472 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:54,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:53:54,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:53:54,473 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:54,473 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:53:54,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:54,477 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/A/4d6f3b70db424f4a84619f347aedb87b as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/4d6f3b70db424f4a84619f347aedb87b 2023-04-19 20:53:54,479 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:54,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:53:54,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:54,485 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:53:54,490 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/4d6f3b70db424f4a84619f347aedb87b, entries=200, sequenceid=683, filesize=14.3 K 2023-04-19 20:53:54,491 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:53:54,491 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:53:54,491 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:53:54,491 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:53:54,491 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:53:54,492 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/B/b7a46c57b6be4fc1bbf2ec6393740cbb as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/b7a46c57b6be4fc1bbf2ec6393740cbb 2023-04-19 20:53:54,492 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:53:54,492 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:53:54,492 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. 2023-04-19 20:53:54,493 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. started... 2023-04-19 20:53:54,493 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. as already flushing 2023-04-19 20:53:54,493 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. 2023-04-19 20:53:54,493 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:53:54,493 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:53:54,493 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:54,493 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:53:54,493 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:53:54,495 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:54,495 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:54,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:54,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:53:54,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:53:54,496 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:54,496 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:53:54,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:53:54,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:53:54,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:54,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:54,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:53:54,498 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:53:54,498 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5b6edd57[Count = 0] remaining members to acquire global barrier 2023-04-19 20:53:54,498 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:53:54,498 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:54,499 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/b7a46c57b6be4fc1bbf2ec6393740cbb, entries=150, sequenceid=683, filesize=11.9 K 2023-04-19 20:53:54,500 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:54,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:54,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:54,501 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:54,501 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:53:54,501 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:53:54,501 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:53:54,501 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:53:54,504 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:54,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:54,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:53:54,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:53:54,505 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:53:54,505 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:53:54,505 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:53:54,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:53:54,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:53:54,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:54,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:54,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:53:54,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:54,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:54,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:53:54,510 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:53:54,510 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:53:54,510 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:53:54,510 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:53:54,510 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:53:54,510 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:53:54,511 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/C/34214c7d26fa4e4082a99c85a1f29782 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/34214c7d26fa4e4082a99c85a1f29782 2023-04-19 20:53:54,516 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:54,516 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:53:54,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:54,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:54,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:53:54,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:53:54,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:54,517 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:54,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:54,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:53:54,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:53:54,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:53:54,519 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:54,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:54,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:53:54,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:54,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:54,520 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:54,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:53:54,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:54,522 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/34214c7d26fa4e4082a99c85a1f29782, entries=150, sequenceid=683, filesize=11.9 K 2023-04-19 20:53:54,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:54,523 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~390.80 KB/400176, currentSize=46.96 KB/48090 for f70bfe945d0054ce5a228cf255d902f5 in 633ms, sequenceid=683, compaction requested=false 2023-04-19 20:53:54,523 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for f70bfe945d0054ce5a228cf255d902f5: 2023-04-19 20:53:54,525 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:53:54,525 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:53:54,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:53:54,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:53:54,526 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:54,526 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:54,526 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:54,526 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:54,526 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:54,526 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:53:54,526 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:53:54,526 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:54,526 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:54,526 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:53:54,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:53:54,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:53:54,526 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:53:54,526 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:53:54,527 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:53:54,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(9158): Flush requested on f70bfe945d0054ce5a228cf255d902f5 2023-04-19 20:53:54,529 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing f70bfe945d0054ce5a228cf255d902f5 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-19 20:53:54,531 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f70bfe945d0054ce5a228cf255d902f5, store=A 2023-04-19 20:53:54,531 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:53:54,531 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f70bfe945d0054ce5a228cf255d902f5, store=B 2023-04-19 20:53:54,531 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:53:54,531 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f70bfe945d0054ce5a228cf255d902f5, store=C 2023-04-19 20:53:54,532 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:53:54,581 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:54,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42350 deadline: 1681937694552, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:54,582 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1681937634548, max timestamp=1681937634553Num uniques 859; 2023-04-19 20:53:54,583 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-19 20:53:54,584 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1681937634548, max timestamp=1681937634553Num uniques 859; 2023-04-19 20:53:54,584 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-19 20:53:54,585 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1681937634548, max timestamp=1681937634553Num uniques 859; 2023-04-19 20:53:54,585 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-19 20:53:54,605 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:54,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 232 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42366 deadline: 1681937694603, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:54,606 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:54,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 241 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42380 deadline: 1681937694603, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:54,606 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:54,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 247 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42342 deadline: 1681937694605, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:54,627 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:53:54,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:53:54,635 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:53:54,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:53:54,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:53:54,664 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:53:54,664 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:53:54,664 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:53:54,664 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:53:54,665 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:54,665 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:54,672 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:53:54,672 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:53:54,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:53:54,672 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:54,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:54,673 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:54,673 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:53:54,673 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:54,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:53:54,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:54,674 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:53:54,684 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:54,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42350 deadline: 1681937694683, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:54,688 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:53:54,688 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:53:54,688 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:53:54,688 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:53:54,688 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:53:54,689 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:53:54,690 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:53:54,690 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. 2023-04-19 20:53:54,690 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. started... 2023-04-19 20:53:54,690 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. as already flushing 2023-04-19 20:53:54,690 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. 2023-04-19 20:53:54,690 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:53:54,690 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:53:54,690 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:54,690 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:53:54,690 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:53:54,701 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:54,702 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:54,702 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:53:54,702 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:54,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:54,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:53:54,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:53:54,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:53:54,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:53:54,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:54,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:54,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:53:54,704 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:53:54,704 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@79eaa6b3[Count = 0] remaining members to acquire global barrier 2023-04-19 20:53:54,704 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:53:54,704 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:54,707 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:54,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:54,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:54,708 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:53:54,708 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:53:54,708 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:53:54,708 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:54,708 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:53:54,710 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:54,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42366 deadline: 1681937694707, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:54,710 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:54,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 243 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42380 deadline: 1681937694707, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:54,710 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:54,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 249 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42342 deadline: 1681937694708, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:54,713 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:54,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:54,713 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:53:54,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:53:54,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:53:54,713 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:53:54,713 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:53:54,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:53:54,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:53:54,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:54,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:54,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:53:54,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:54,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:54,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:53:54,720 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:53:54,721 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:53:54,721 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:53:54,721 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:53:54,721 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:53:54,721 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:53:54,723 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:54,723 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:54,723 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:54,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:54,723 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:53:54,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:54,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:53:54,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:53:54,724 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:54,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:53:54,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:53:54,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:54,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:53:54,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:54,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:53:54,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:54,736 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:54,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:54,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:53:54,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:54,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:54,742 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:54,742 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:54,742 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:54,742 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:54,742 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:54,742 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:54,742 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:54,742 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:53:54,742 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:53:54,742 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:53:54,742 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:53:54,742 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:53:54,742 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:53:54,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:53:54,742 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:53:54,742 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:53:54,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:53:54,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:53:54,843 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:53:54,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:53:54,845 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:53:54,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:53:54,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:53:54,847 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:53:54,847 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:53:54,847 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:53:54,847 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:53:54,847 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:54,847 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:54,849 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:53:54,849 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:54,849 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:53:54,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:53:54,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:54,849 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:54,850 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:53:54,850 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:54,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:53:54,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:54,850 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:53:54,850 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:53:54,850 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:53:54,850 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:53:54,851 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:53:54,851 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:53:54,851 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:53:54,851 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:53:54,851 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. 2023-04-19 20:53:54,851 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. started... 2023-04-19 20:53:54,851 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. as already flushing 2023-04-19 20:53:54,851 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. 2023-04-19 20:53:54,851 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:53:54,851 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:53:54,851 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:54,851 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:53:54,851 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:53:54,853 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:54,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:54,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:53:54,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:53:54,853 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:54,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:53:54,853 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:54,853 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:53:54,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:53:54,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:54,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:54,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:53:54,855 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:53:54,855 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:53:54,855 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@347f5c80[Count = 0] remaining members to acquire global barrier 2023-04-19 20:53:54,855 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:54,856 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:54,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:54,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:54,857 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:53:54,857 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:53:54,857 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:53:54,857 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:54,857 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:53:54,860 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:53:54,860 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:54,860 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:53:54,860 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:53:54,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:54,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:53:54,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:53:54,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:53:54,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:53:54,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:54,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:54,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:53:54,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:54,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:54,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:53:54,865 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:53:54,865 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:53:54,865 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:53:54,865 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:53:54,865 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:53:54,865 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:53:54,867 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:54,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:54,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:53:54,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:53:54,868 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:54,868 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:53:54,868 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:54,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:54,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:53:54,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:53:54,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:53:54,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:54,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:53:54,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:54,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:54,870 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:54,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:54,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:53:54,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:54,872 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:54,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:54,877 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:53:54,877 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:53:54,878 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:54,878 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:54,878 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:54,878 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:53:54,878 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:53:54,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:53:54,878 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:53:54,878 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:53:54,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:53:54,878 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:54,878 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:54,878 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:54,878 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:54,878 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:53:54,879 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:53:54,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:53:54,886 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:54,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42350 deadline: 1681937694885, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:54,913 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:54,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42366 deadline: 1681937694911, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:54,913 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:54,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 245 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42380 deadline: 1681937694912, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:54,915 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:54,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 251 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42342 deadline: 1681937694912, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:54,971 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=698 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/A/9b3d5fd0cd044e248b6db6802c4ae0d4 2023-04-19 20:53:54,978 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:53:54,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:53:54,982 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:53:54,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:53:54,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:53:54,984 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:53:54,984 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:53:54,984 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:53:54,984 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:53:54,985 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:54,985 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:54,986 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:53:54,986 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:54,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:53:54,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:53:54,987 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:54,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:54,987 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:53:54,987 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:54,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:53:54,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:54,988 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:53:54,988 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:53:54,988 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:53:54,988 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:53:54,988 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:53:54,988 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:53:54,988 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:53:54,988 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:53:54,988 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. 2023-04-19 20:53:54,988 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. started... 2023-04-19 20:53:54,988 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. as already flushing 2023-04-19 20:53:54,988 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. 2023-04-19 20:53:54,988 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:53:54,989 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:53:54,989 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:54,989 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:53:54,989 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:53:54,991 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:54,991 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:54,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:54,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:53:54,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:53:54,991 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:54,991 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:53:54,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:53:54,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:53:54,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:54,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:54,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:53:54,993 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:53:54,993 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3c684c67[Count = 0] remaining members to acquire global barrier 2023-04-19 20:53:54,993 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:53:54,993 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:54,994 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:54,994 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:53:54,995 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:54,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:54,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:54,995 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:53:54,995 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:53:54,995 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:53:54,997 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:53:54,997 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:53:54,997 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:53:54,997 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:54,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:54,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:53:54,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:53:54,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:53:54,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:53:54,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:54,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:54,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:53:54,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:54,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:55,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:53:55,000 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:53:55,000 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:53:55,000 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:53:55,000 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:53:55,000 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:53:55,000 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:53:55,003 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:55,003 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:55,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:55,003 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:53:55,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:55,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:53:55,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:53:55,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:55,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:53:55,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:53:55,004 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:55,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:55,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:53:55,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:55,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:53:55,005 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:55,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:55,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:55,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:53:55,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:55,009 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:53:55,009 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:53:55,009 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:53:55,009 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:53:55,009 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:53:55,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:53:55,009 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:55,010 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:55,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:53:55,010 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:53:55,010 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:55,010 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:55,010 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:53:55,010 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:55,010 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:55,010 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:55,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:53:55,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:53:55,010 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:53:55,110 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:53:55,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:53:55,112 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:53:55,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:53:55,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:53:55,113 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:53:55,114 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:53:55,114 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:53:55,114 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:53:55,114 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:55,114 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:55,116 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:53:55,116 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:55,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:53:55,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:53:55,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:55,116 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:55,116 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:53:55,117 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:55,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:53:55,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:55,117 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:53:55,117 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:53:55,117 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:53:55,118 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:53:55,118 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:53:55,118 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:53:55,118 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:53:55,118 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:53:55,118 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. 2023-04-19 20:53:55,118 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. started... 2023-04-19 20:53:55,118 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. as already flushing 2023-04-19 20:53:55,118 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. 2023-04-19 20:53:55,118 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:53:55,118 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:53:55,118 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:55,118 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:53:55,118 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:53:55,120 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:55,120 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:55,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:55,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:53:55,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:53:55,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:53:55,133 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:55,133 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:53:55,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:53:55,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:55,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:55,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:53:55,135 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:53:55,136 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1141bad8[Count = 0] remaining members to acquire global barrier 2023-04-19 20:53:55,136 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:53:55,136 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:55,137 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:55,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:55,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:55,137 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:53:55,137 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:53:55,137 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:53:55,137 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:55,138 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:53:55,139 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:55,139 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:53:55,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:55,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:53:55,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:53:55,140 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:53:55,140 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:53:55,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:53:55,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:53:55,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:55,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:55,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:53:55,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:55,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:55,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:53:55,142 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:53:55,142 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:53:55,142 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:53:55,142 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:53:55,142 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:53:55,142 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:53:55,145 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:55,145 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:55,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:55,145 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:53:55,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:55,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:55,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:53:55,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:53:55,146 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:53:55,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:53:55,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:53:55,146 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:55,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:55,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:55,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:53:55,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:55,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:55,147 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:55,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:53:55,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:55,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:55,160 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:55,160 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:53:55,160 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:55,160 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:53:55,160 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:55,160 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:53:55,160 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:53:55,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:53:55,160 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:53:55,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:53:55,160 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:53:55,160 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:55,161 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:53:55,161 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:55,161 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:53:55,161 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:55,161 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:55,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:53:55,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:53:55,188 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:55,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42350 deadline: 1681937695188, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:55,218 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:55,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42366 deadline: 1681937695215, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:55,218 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:55,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 247 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42380 deadline: 1681937695217, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:55,218 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:55,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 253 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42342 deadline: 1681937695218, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:55,261 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:53:55,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:53:55,263 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:53:55,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:53:55,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:53:55,265 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:53:55,265 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:53:55,265 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:53:55,265 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:53:55,265 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:55,265 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:55,267 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:53:55,267 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:53:55,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:53:55,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:55,268 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:55,268 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:55,268 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:53:55,268 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:55,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:53:55,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:55,269 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:53:55,269 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:53:55,269 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:53:55,269 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:53:55,269 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:53:55,269 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:53:55,270 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:53:55,270 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:53:55,270 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. 2023-04-19 20:53:55,270 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. started... 2023-04-19 20:53:55,270 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. as already flushing 2023-04-19 20:53:55,270 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. 2023-04-19 20:53:55,270 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:53:55,270 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:53:55,270 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:55,270 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:53:55,270 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:53:55,272 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:55,272 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:55,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:55,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:53:55,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:53:55,272 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:55,272 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:53:55,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:53:55,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:53:55,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:55,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:55,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:53:55,274 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:53:55,274 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3db2597d[Count = 0] remaining members to acquire global barrier 2023-04-19 20:53:55,274 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:53:55,274 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:55,276 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:55,276 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:55,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:55,276 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:53:55,276 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:53:55,277 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:53:55,277 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:55,277 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:53:55,278 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:55,278 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:53:55,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:55,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:53:55,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:53:55,278 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:53:55,278 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:53:55,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:53:55,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:53:55,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:55,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:55,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:53:55,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:55,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:55,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:53:55,280 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:53:55,280 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:53:55,280 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:53:55,280 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:53:55,280 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:53:55,280 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:53:55,282 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:55,282 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:53:55,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:55,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:55,282 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:55,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:55,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:53:55,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:53:55,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:53:55,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:53:55,283 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:55,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:53:55,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:55,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:55,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:53:55,284 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:55,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:55,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:55,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:53:55,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:55,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:55,286 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:53:55,286 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:53:55,286 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:55,286 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:55,286 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:55,286 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:55,286 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:55,286 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:55,286 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:55,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:53:55,286 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:53:55,286 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:53:55,286 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:53:55,286 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:53:55,287 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:53:55,286 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:53:55,287 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:53:55,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:53:55,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:53:55,386 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=698 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/B/f47c98c5c223486a9d083bfdf20de744 2023-04-19 20:53:55,387 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:53:55,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:53:55,398 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:53:55,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:53:55,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:53:55,401 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:53:55,401 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:53:55,401 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:53:55,401 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:53:55,401 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:55,401 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:55,403 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:53:55,404 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:55,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:53:55,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:53:55,404 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:55,404 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:53:55,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:55,404 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:55,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:53:55,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:55,405 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:53:55,409 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:53:55,409 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:53:55,409 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:53:55,409 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:53:55,409 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:53:55,420 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:53:55,420 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:53:55,420 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. 2023-04-19 20:53:55,420 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. started... 2023-04-19 20:53:55,420 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. as already flushing 2023-04-19 20:53:55,420 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. 2023-04-19 20:53:55,420 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:53:55,420 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:53:55,420 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:55,421 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:53:55,421 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:53:55,422 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:55,422 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:55,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:55,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:53:55,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:53:55,423 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:55,423 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:53:55,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:53:55,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:53:55,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:55,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:55,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:53:55,424 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:53:55,425 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@73679917[Count = 0] remaining members to acquire global barrier 2023-04-19 20:53:55,425 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:53:55,425 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:55,426 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:55,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:55,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:55,426 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:53:55,426 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:53:55,426 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:53:55,427 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:55,427 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:53:55,429 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:55,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:55,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:53:55,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:53:55,429 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:53:55,429 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:53:55,429 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:53:55,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:53:55,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:53:55,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:55,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:55,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:53:55,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:55,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:55,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:53:55,432 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:53:55,432 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:53:55,432 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:53:55,432 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:53:55,432 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:53:55,432 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:53:55,434 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=698 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/C/9c7eb29c72d243b2821225121fba85ba 2023-04-19 20:53:55,435 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:55,435 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:53:55,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:55,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:55,435 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:55,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:55,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:53:55,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:53:55,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:53:55,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:53:55,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:55,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:53:55,436 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:55,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:55,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:53:55,439 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:55,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:55,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:55,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:53:55,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:55,442 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/A/9b3d5fd0cd044e248b6db6802c4ae0d4 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/9b3d5fd0cd044e248b6db6802c4ae0d4 2023-04-19 20:53:55,443 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:53:55,443 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:53:55,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:53:55,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:53:55,444 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:55,444 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:55,444 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:55,444 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:55,444 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:55,444 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:55,444 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:53:55,444 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:53:55,444 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:53:55,444 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:55,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:53:55,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:53:55,444 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:53:55,444 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:53:55,445 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:53:55,448 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/9b3d5fd0cd044e248b6db6802c4ae0d4, entries=150, sequenceid=698, filesize=11.9 K 2023-04-19 20:53:55,449 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/B/f47c98c5c223486a9d083bfdf20de744 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/f47c98c5c223486a9d083bfdf20de744 2023-04-19 20:53:55,455 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/f47c98c5c223486a9d083bfdf20de744, entries=150, sequenceid=698, filesize=11.9 K 2023-04-19 20:53:55,456 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/C/9c7eb29c72d243b2821225121fba85ba as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/9c7eb29c72d243b2821225121fba85ba 2023-04-19 20:53:55,462 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/9c7eb29c72d243b2821225121fba85ba, entries=150, sequenceid=698, filesize=11.9 K 2023-04-19 20:53:55,463 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=120.76 KB/123660 for f70bfe945d0054ce5a228cf255d902f5 in 934ms, sequenceid=698, compaction requested=true 2023-04-19 20:53:55,463 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for f70bfe945d0054ce5a228cf255d902f5: 2023-04-19 20:53:55,463 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 20:53:55,463 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-19 20:53:55,463 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 20:53:55,463 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-19 20:53:55,464 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-19 20:53:55,465 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39544 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-19 20:53:55,465 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1912): f70bfe945d0054ce5a228cf255d902f5/A is initiating minor compaction (all files) 2023-04-19 20:53:55,465 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of f70bfe945d0054ce5a228cf255d902f5/A in TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. 2023-04-19 20:53:55,465 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/78780e4b985840c18ce9ab9115c522dd, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/4d6f3b70db424f4a84619f347aedb87b, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/9b3d5fd0cd044e248b6db6802c4ae0d4] into tmpdir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp, totalSize=38.6 K 2023-04-19 20:53:55,465 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37104 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-19 20:53:55,465 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1912): f70bfe945d0054ce5a228cf255d902f5/B is initiating minor compaction (all files) 2023-04-19 20:53:55,465 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2259): Starting compaction of f70bfe945d0054ce5a228cf255d902f5/B in TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. 2023-04-19 20:53:55,466 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/e7b5f69e860f4c1aa8d50776099e1538, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/b7a46c57b6be4fc1bbf2ec6393740cbb, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/f47c98c5c223486a9d083bfdf20de744] into tmpdir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp, totalSize=36.2 K 2023-04-19 20:53:55,466 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 78780e4b985840c18ce9ab9115c522dd, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=648, earliestPutTs=1681937633680 2023-04-19 20:53:55,467 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting e7b5f69e860f4c1aa8d50776099e1538, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=648, earliestPutTs=1681937633680 2023-04-19 20:53:55,469 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 4d6f3b70db424f4a84619f347aedb87b, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=683, earliestPutTs=1681937633771 2023-04-19 20:53:55,469 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting b7a46c57b6be4fc1bbf2ec6393740cbb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=683, earliestPutTs=1681937633771 2023-04-19 20:53:55,470 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 9b3d5fd0cd044e248b6db6802c4ae0d4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=698, earliestPutTs=1681937633902 2023-04-19 20:53:55,470 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting f47c98c5c223486a9d083bfdf20de744, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=698, earliestPutTs=1681937633902 2023-04-19 20:53:55,479 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] throttle.PressureAwareThroughputController(145): f70bfe945d0054ce5a228cf255d902f5#A#compaction#57 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 20:53:55,481 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] throttle.PressureAwareThroughputController(145): f70bfe945d0054ce5a228cf255d902f5#B#compaction#58 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 20:53:55,503 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/B/c09cb8f541374249b000d1974a826a55 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/c09cb8f541374249b000d1974a826a55 2023-04-19 20:53:55,511 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in f70bfe945d0054ce5a228cf255d902f5/B of f70bfe945d0054ce5a228cf255d902f5 into c09cb8f541374249b000d1974a826a55(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 20:53:55,511 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for f70bfe945d0054ce5a228cf255d902f5: 2023-04-19 20:53:55,511 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5., storeName=f70bfe945d0054ce5a228cf255d902f5/B, priority=13, startTime=1681937635463; duration=0sec 2023-04-19 20:53:55,511 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 20:53:55,512 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-19 20:53:55,513 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37104 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-19 20:53:55,513 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1912): f70bfe945d0054ce5a228cf255d902f5/C is initiating minor compaction (all files) 2023-04-19 20:53:55,513 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2259): Starting compaction of f70bfe945d0054ce5a228cf255d902f5/C in TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. 2023-04-19 20:53:55,513 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/fc84d90f0d544b1dab4c05b7edfd112f, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/34214c7d26fa4e4082a99c85a1f29782, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/9c7eb29c72d243b2821225121fba85ba] into tmpdir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp, totalSize=36.2 K 2023-04-19 20:53:55,514 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting fc84d90f0d544b1dab4c05b7edfd112f, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=648, earliestPutTs=1681937633680 2023-04-19 20:53:55,514 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 34214c7d26fa4e4082a99c85a1f29782, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=683, earliestPutTs=1681937633771 2023-04-19 20:53:55,516 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 9c7eb29c72d243b2821225121fba85ba, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=698, earliestPutTs=1681937633902 2023-04-19 20:53:55,525 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] throttle.PressureAwareThroughputController(145): f70bfe945d0054ce5a228cf255d902f5#C#compaction#59 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 20:53:55,541 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/C/9407d1254a5c43de9a87022d7daf7a22 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/9407d1254a5c43de9a87022d7daf7a22 2023-04-19 20:53:55,544 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:53:55,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:53:55,550 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:53:55,551 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in f70bfe945d0054ce5a228cf255d902f5/C of f70bfe945d0054ce5a228cf255d902f5 into 9407d1254a5c43de9a87022d7daf7a22(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 20:53:55,551 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for f70bfe945d0054ce5a228cf255d902f5: 2023-04-19 20:53:55,551 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5., storeName=f70bfe945d0054ce5a228cf255d902f5/C, priority=13, startTime=1681937635463; duration=0sec 2023-04-19 20:53:55,551 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 20:53:55,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:53:55,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:53:55,552 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:53:55,552 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:53:55,552 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:53:55,552 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:53:55,552 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:55,553 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:55,554 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:53:55,554 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:55,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:53:55,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:53:55,554 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:55,554 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:53:55,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:55,555 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:55,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:53:55,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:55,555 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:53:55,555 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:53:55,555 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:53:55,555 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:53:55,555 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:53:55,555 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:53:55,556 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:53:55,556 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:53:55,556 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. 2023-04-19 20:53:55,556 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. started... 2023-04-19 20:53:55,556 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing f70bfe945d0054ce5a228cf255d902f5 3/3 column families, dataSize=120.76 KB heapSize=340.45 KB 2023-04-19 20:53:55,556 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f70bfe945d0054ce5a228cf255d902f5, store=A 2023-04-19 20:53:55,556 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 20:53:55,556 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f70bfe945d0054ce5a228cf255d902f5, store=B 2023-04-19 20:53:55,556 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 20:53:55,556 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f70bfe945d0054ce5a228cf255d902f5, store=C 2023-04-19 20:53:55,556 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 20:53:55,570 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=734 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/A/044073d919ed4b08908d57b46c1885e0 2023-04-19 20:53:55,605 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=734 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/B/5949adbab09645cfa3cf825e42a09599 2023-04-19 20:53:55,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(9158): Flush requested on f70bfe945d0054ce5a228cf255d902f5 2023-04-19 20:53:55,695 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. as already flushing 2023-04-19 20:53:55,719 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:55,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42350 deadline: 1681937695716, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:55,723 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:55,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42366 deadline: 1681937695722, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:55,724 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:55,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 249 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42380 deadline: 1681937695723, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:55,728 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:55,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 255 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42342 deadline: 1681937695723, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:55,823 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:55,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42350 deadline: 1681937695820, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:55,905 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/A/b0fe4da425c746a193066d70af799d1a as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/b0fe4da425c746a193066d70af799d1a 2023-04-19 20:53:55,912 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in f70bfe945d0054ce5a228cf255d902f5/A of f70bfe945d0054ce5a228cf255d902f5 into b0fe4da425c746a193066d70af799d1a(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 20:53:55,912 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for f70bfe945d0054ce5a228cf255d902f5: 2023-04-19 20:53:55,912 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5., storeName=f70bfe945d0054ce5a228cf255d902f5/A, priority=13, startTime=1681937635463; duration=0sec 2023-04-19 20:53:55,912 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 20:53:56,023 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=734 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/C/fabfaf48242b46048e1d5ab2cd87c458 2023-04-19 20:53:56,026 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:56,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 233 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42350 deadline: 1681937696025, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:56,032 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/A/044073d919ed4b08908d57b46c1885e0 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/044073d919ed4b08908d57b46c1885e0 2023-04-19 20:53:56,039 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/044073d919ed4b08908d57b46c1885e0, entries=150, sequenceid=734, filesize=11.9 K 2023-04-19 20:53:56,041 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/B/5949adbab09645cfa3cf825e42a09599 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/5949adbab09645cfa3cf825e42a09599 2023-04-19 20:53:56,052 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/5949adbab09645cfa3cf825e42a09599, entries=150, sequenceid=734, filesize=11.9 K 2023-04-19 20:53:56,057 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/C/fabfaf48242b46048e1d5ab2cd87c458 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/fabfaf48242b46048e1d5ab2cd87c458 2023-04-19 20:53:56,075 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/fabfaf48242b46048e1d5ab2cd87c458, entries=150, sequenceid=734, filesize=11.9 K 2023-04-19 20:53:56,076 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~340.41 KB/348576, currentSize=60.38 KB/61830 for f70bfe945d0054ce5a228cf255d902f5 in 520ms, sequenceid=734, compaction requested=false 2023-04-19 20:53:56,076 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f70bfe945d0054ce5a228cf255d902f5: 2023-04-19 20:53:56,076 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. 2023-04-19 20:53:56,077 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:53:56,077 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:53:56,077 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:56,084 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:53:56,084 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:53:56,089 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:56,089 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:56,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:56,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:53:56,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:53:56,090 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:56,090 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:53:56,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:53:56,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:53:56,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:56,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:56,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:53:56,092 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:53:56,092 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@d92e868[Count = 0] remaining members to acquire global barrier 2023-04-19 20:53:56,092 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:53:56,092 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:56,095 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:56,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:56,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:56,095 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:53:56,095 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:53:56,095 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:53:56,097 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:56,097 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:53:56,099 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:56,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:56,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:53:56,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:53:56,100 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:53:56,100 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:53:56,100 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:53:56,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:53:56,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:53:56,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:56,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:56,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:53:56,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:56,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:56,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:53:56,106 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:53:56,106 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:53:56,106 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:53:56,106 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:53:56,107 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:53:56,107 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:53:56,110 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:56,110 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:53:56,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:56,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:56,110 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:56,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:56,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:53:56,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:53:56,111 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:53:56,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:53:56,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:53:56,111 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:56,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:56,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:56,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:53:56,113 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:56,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:56,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:56,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:53:56,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:56,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:56,122 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:56,122 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:56,122 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:56,122 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:56,122 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:56,122 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:56,122 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:56,122 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:53:56,122 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:53:56,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:53:56,123 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:53:56,123 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:53:56,123 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:53:56,123 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:53:56,123 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:53:56,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:53:56,123 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:53:56,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:53:56,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:53:56,223 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:53:56,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:53:56,227 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:53:56,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:53:56,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:53:56,238 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:53:56,238 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:53:56,238 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:53:56,238 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:53:56,239 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:56,239 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:56,241 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:56,241 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:53:56,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:53:56,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:53:56,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:56,242 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:56,242 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:53:56,242 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:56,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:53:56,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:56,243 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:53:56,248 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:53:56,248 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:53:56,248 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:53:56,248 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:53:56,248 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:53:56,249 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:53:56,249 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:53:56,249 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. 2023-04-19 20:53:56,249 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. started... 2023-04-19 20:53:56,249 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing f70bfe945d0054ce5a228cf255d902f5 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-19 20:53:56,250 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f70bfe945d0054ce5a228cf255d902f5, store=A 2023-04-19 20:53:56,250 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:53:56,250 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f70bfe945d0054ce5a228cf255d902f5, store=B 2023-04-19 20:53:56,250 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:53:56,250 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f70bfe945d0054ce5a228cf255d902f5, store=C 2023-04-19 20:53:56,250 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:53:56,285 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=747 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/A/7f20fbd572a54adc92250879c385c67e 2023-04-19 20:53:56,319 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=747 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/B/6f60ac93c4f349848df99b8fa81dc88a 2023-04-19 20:53:56,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(9158): Flush requested on f70bfe945d0054ce5a228cf255d902f5 2023-04-19 20:53:56,331 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. as already flushing 2023-04-19 20:53:56,397 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:56,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 252 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42350 deadline: 1681937696394, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:56,502 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:56,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 254 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42350 deadline: 1681937696499, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:56,706 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:56,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 256 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42350 deadline: 1681937696704, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:56,727 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:56,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 251 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42380 deadline: 1681937696727, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:56,732 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:56,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 242 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42366 deadline: 1681937696731, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:56,732 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:56,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 257 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42342 deadline: 1681937696731, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:56,774 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=747 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/C/182d5ab2d07a4cae817fe22555780f69 2023-04-19 20:53:56,781 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/A/7f20fbd572a54adc92250879c385c67e as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/7f20fbd572a54adc92250879c385c67e 2023-04-19 20:53:56,787 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/7f20fbd572a54adc92250879c385c67e, entries=150, sequenceid=747, filesize=11.9 K 2023-04-19 20:53:56,790 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/B/6f60ac93c4f349848df99b8fa81dc88a as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/6f60ac93c4f349848df99b8fa81dc88a 2023-04-19 20:53:56,799 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/6f60ac93c4f349848df99b8fa81dc88a, entries=150, sequenceid=747, filesize=11.9 K 2023-04-19 20:53:56,800 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/C/182d5ab2d07a4cae817fe22555780f69 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/182d5ab2d07a4cae817fe22555780f69 2023-04-19 20:53:56,811 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/182d5ab2d07a4cae817fe22555780f69, entries=150, sequenceid=747, filesize=11.9 K 2023-04-19 20:53:56,812 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for f70bfe945d0054ce5a228cf255d902f5 in 563ms, sequenceid=747, compaction requested=true 2023-04-19 20:53:56,812 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f70bfe945d0054ce5a228cf255d902f5: 2023-04-19 20:53:56,812 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. 2023-04-19 20:53:56,812 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:53:56,812 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:53:56,812 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:56,812 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:53:56,812 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:53:56,814 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:56,814 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:56,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:56,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:53:56,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:53:56,814 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:56,814 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:53:56,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:53:56,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:53:56,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:56,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:56,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:53:56,816 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:53:56,816 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@11191478[Count = 0] remaining members to acquire global barrier 2023-04-19 20:53:56,816 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:53:56,816 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:56,818 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:56,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:56,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:56,818 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:53:56,818 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:53:56,818 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:53:56,818 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:56,818 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:53:56,820 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:53:56,820 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:53:56,820 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:53:56,820 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:56,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:56,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:53:56,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:53:56,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:53:56,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:53:56,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:56,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:56,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:53:56,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:56,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:56,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:53:56,823 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:53:56,823 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:53:56,823 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:53:56,823 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:53:56,823 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:53:56,823 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:53:56,829 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:56,829 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:53:56,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:56,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:56,829 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:56,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:56,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:53:56,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:53:56,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:53:56,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:53:56,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:53:56,830 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:56,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:56,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:56,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:53:56,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:56,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:56,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:53:56,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:56,834 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:56,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:56,842 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:53:56,842 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:56,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:53:56,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:53:56,843 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:53:56,843 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:53:56,843 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:56,843 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:56,843 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:56,843 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:56,843 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:56,843 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:56,843 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:53:56,842 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:53:56,843 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:53:56,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:53:56,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:53:56,843 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:53:56,943 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:53:56,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:53:56,945 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:53:56,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:53:56,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:53:56,947 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:53:56,947 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:53:56,947 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:53:56,947 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:53:56,947 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:56,947 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:56,949 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:56,949 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:53:56,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:53:56,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:53:56,949 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:56,949 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:53:56,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:56,949 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:56,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:53:56,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:56,950 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:53:56,950 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:53:56,950 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:53:56,950 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:53:56,950 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:53:56,950 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:53:56,950 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:53:56,950 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:53:56,950 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. 2023-04-19 20:53:56,951 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. started... 2023-04-19 20:53:56,951 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing f70bfe945d0054ce5a228cf255d902f5 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-19 20:53:56,951 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f70bfe945d0054ce5a228cf255d902f5, store=A 2023-04-19 20:53:56,951 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:53:56,951 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f70bfe945d0054ce5a228cf255d902f5, store=B 2023-04-19 20:53:56,951 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:53:56,951 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f70bfe945d0054ce5a228cf255d902f5, store=C 2023-04-19 20:53:56,951 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:53:57,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(9158): Flush requested on f70bfe945d0054ce5a228cf255d902f5 2023-04-19 20:53:57,011 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. as already flushing 2023-04-19 20:53:57,030 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:57,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 267 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42350 deadline: 1681937697028, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:57,136 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:57,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 269 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42350 deadline: 1681937697131, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:57,341 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:57,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 271 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42350 deadline: 1681937697338, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:57,366 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=767 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/A/9890432f65ab43ec80ca6142349e6f56 2023-04-19 20:53:57,380 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=767 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/B/62282af0f5144651a6b3d54b7c733b36 2023-04-19 20:53:57,644 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:57,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 273 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42350 deadline: 1681937697643, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:57,796 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=767 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/C/6faed4d004564a5da0f1773f2b8fc1b6 2023-04-19 20:53:57,805 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/A/9890432f65ab43ec80ca6142349e6f56 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/9890432f65ab43ec80ca6142349e6f56 2023-04-19 20:53:57,811 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/9890432f65ab43ec80ca6142349e6f56, entries=150, sequenceid=767, filesize=11.9 K 2023-04-19 20:53:57,812 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/B/62282af0f5144651a6b3d54b7c733b36 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/62282af0f5144651a6b3d54b7c733b36 2023-04-19 20:53:57,818 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/62282af0f5144651a6b3d54b7c733b36, entries=150, sequenceid=767, filesize=11.9 K 2023-04-19 20:53:57,819 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/C/6faed4d004564a5da0f1773f2b8fc1b6 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/6faed4d004564a5da0f1773f2b8fc1b6 2023-04-19 20:53:57,824 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/6faed4d004564a5da0f1773f2b8fc1b6, entries=150, sequenceid=767, filesize=11.9 K 2023-04-19 20:53:57,825 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for f70bfe945d0054ce5a228cf255d902f5 in 874ms, sequenceid=767, compaction requested=true 2023-04-19 20:53:57,825 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f70bfe945d0054ce5a228cf255d902f5: 2023-04-19 20:53:57,826 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. 2023-04-19 20:53:57,826 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:53:57,826 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:53:57,826 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:57,826 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:53:57,826 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:53:57,834 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:57,834 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:57,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:57,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:53:57,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:53:57,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:53:57,839 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:57,839 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:53:57,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:53:57,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:57,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:57,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:53:57,841 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:53:57,841 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3ecfce97[Count = 0] remaining members to acquire global barrier 2023-04-19 20:53:57,841 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:53:57,841 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:57,843 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:57,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:57,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:57,844 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:53:57,844 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:53:57,844 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:53:57,844 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:57,844 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:53:57,845 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:53:57,846 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:53:57,846 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:57,846 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:53:57,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:57,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:53:57,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:53:57,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:53:57,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:53:57,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:57,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:57,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:53:57,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:57,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:57,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:53:57,849 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:53:57,849 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:53:57,849 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:53:57,849 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:53:57,849 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:53:57,849 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:53:57,851 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:57,851 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:53:57,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:57,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:57,851 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:57,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:53:57,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:53:57,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:57,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:53:57,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:53:57,852 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:57,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:57,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:53:57,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:57,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:53:57,853 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:57,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:57,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:57,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:53:57,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:57,856 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:53:57,856 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:53:57,856 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:57,856 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:57,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:53:57,856 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:53:57,856 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:57,856 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:57,856 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:57,856 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:57,856 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:57,856 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:53:57,856 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:53:57,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:53:57,857 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:53:57,857 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:53:57,857 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:53:57,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:53:57,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:53:57,957 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:53:57,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:53:57,960 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:53:57,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:53:57,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:53:57,964 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:53:57,964 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:53:57,964 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:53:57,964 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:53:57,965 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:57,965 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:57,967 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:57,967 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:53:57,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:53:57,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:53:57,968 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:57,968 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:53:57,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:57,969 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:57,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:53:57,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:57,969 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:53:57,969 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:53:57,969 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:53:57,970 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:53:57,970 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:53:57,970 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:53:57,970 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:53:57,970 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:53:57,970 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. 2023-04-19 20:53:57,971 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. started... 2023-04-19 20:53:57,971 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing f70bfe945d0054ce5a228cf255d902f5 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-19 20:53:57,971 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f70bfe945d0054ce5a228cf255d902f5, store=A 2023-04-19 20:53:57,971 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:53:57,971 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f70bfe945d0054ce5a228cf255d902f5, store=B 2023-04-19 20:53:57,971 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:53:57,971 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f70bfe945d0054ce5a228cf255d902f5, store=C 2023-04-19 20:53:57,971 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:53:57,986 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=779 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/A/992f11e8de07463186c620f55d66fbf2 2023-04-19 20:53:58,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(9158): Flush requested on f70bfe945d0054ce5a228cf255d902f5 2023-04-19 20:53:58,150 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. as already flushing 2023-04-19 20:53:58,216 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:58,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 292 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42350 deadline: 1681937698214, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:58,318 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:58,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 294 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42350 deadline: 1681937698318, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:58,403 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=779 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/B/1b535ff8dc764876bdea11aafd523c70 2023-04-19 20:53:58,417 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=779 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/C/3b3520a3aa9a4571b8953d385143b0a6 2023-04-19 20:53:58,424 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/A/992f11e8de07463186c620f55d66fbf2 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/992f11e8de07463186c620f55d66fbf2 2023-04-19 20:53:58,430 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/992f11e8de07463186c620f55d66fbf2, entries=150, sequenceid=779, filesize=11.9 K 2023-04-19 20:53:58,431 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/B/1b535ff8dc764876bdea11aafd523c70 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/1b535ff8dc764876bdea11aafd523c70 2023-04-19 20:53:58,440 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/1b535ff8dc764876bdea11aafd523c70, entries=150, sequenceid=779, filesize=11.9 K 2023-04-19 20:53:58,441 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/C/3b3520a3aa9a4571b8953d385143b0a6 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/3b3520a3aa9a4571b8953d385143b0a6 2023-04-19 20:53:58,447 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/3b3520a3aa9a4571b8953d385143b0a6, entries=150, sequenceid=779, filesize=11.9 K 2023-04-19 20:53:58,448 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for f70bfe945d0054ce5a228cf255d902f5 in 477ms, sequenceid=779, compaction requested=true 2023-04-19 20:53:58,448 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f70bfe945d0054ce5a228cf255d902f5: 2023-04-19 20:53:58,448 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. 2023-04-19 20:53:58,448 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:53:58,448 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:53:58,448 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:58,448 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:53:58,448 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:53:58,450 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:58,450 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:58,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:58,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:53:58,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:53:58,451 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:58,451 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:53:58,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:53:58,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:53:58,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:58,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:58,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:53:58,452 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:53:58,452 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@712f4778[Count = 0] remaining members to acquire global barrier 2023-04-19 20:53:58,452 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:53:58,452 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:58,454 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:58,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:58,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:58,454 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:53:58,454 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:53:58,454 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:53:58,454 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:58,454 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:53:58,456 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:58,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:58,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:53:58,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:53:58,456 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:53:58,456 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:53:58,456 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:53:58,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:53:58,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:53:58,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:58,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:58,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:53:58,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:58,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:58,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:53:58,459 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:53:58,459 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:53:58,459 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:53:58,459 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:53:58,459 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:53:58,459 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:53:58,461 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:58,461 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:58,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:58,461 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:53:58,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:58,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:53:58,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:53:58,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:58,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:53:58,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:53:58,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:53:58,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:58,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:58,463 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:58,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:53:58,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:58,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:58,464 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:58,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:53:58,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:58,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:58,467 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:58,467 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:58,467 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:58,467 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:58,467 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:58,467 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:58,467 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:58,467 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:53:58,467 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:53:58,467 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:53:58,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:53:58,467 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:53:58,467 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:53:58,467 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:53:58,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:53:58,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:53:58,468 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:53:58,468 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:53:58,467 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:53:58,527 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681937638203, max timestamp=1681937638525Num uniques 859; 2023-04-19 20:53:58,528 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 20:53:58,528 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681937638203, max timestamp=1681937638525Num uniques 859; 2023-04-19 20:53:58,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(9158): Flush requested on f70bfe945d0054ce5a228cf255d902f5 2023-04-19 20:53:58,528 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681937638203, max timestamp=1681937638525Num uniques 859; 2023-04-19 20:53:58,528 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 20:53:58,528 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing f70bfe945d0054ce5a228cf255d902f5 3/3 column families, dataSize=58.14 KB heapSize=164.17 KB 2023-04-19 20:53:58,528 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 20:53:58,528 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f70bfe945d0054ce5a228cf255d902f5, store=A 2023-04-19 20:53:58,528 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 20:53:58,528 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f70bfe945d0054ce5a228cf255d902f5, store=B 2023-04-19 20:53:58,529 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 20:53:58,529 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f70bfe945d0054ce5a228cf255d902f5, store=C 2023-04-19 20:53:58,529 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 20:53:58,560 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=800 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/A/ccf5eaa3d0f04dc598d3f0b104e1ffdf 2023-04-19 20:53:58,568 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:53:58,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:53:58,572 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:53:58,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:53:58,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:53:58,575 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:53:58,576 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:53:58,576 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:53:58,576 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:53:58,576 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:58,577 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:58,581 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:58,581 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:53:58,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:53:58,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:53:58,581 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:58,582 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:53:58,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:58,582 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:58,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:53:58,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:58,583 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:53:58,584 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:53:58,584 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:53:58,584 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=800 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/B/15887afc4d5947bd9c14d93ec5e8a98e 2023-04-19 20:53:58,584 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:53:58,584 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:53:58,584 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:53:58,585 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:53:58,585 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:53:58,585 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. 2023-04-19 20:53:58,585 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. started... 2023-04-19 20:53:58,585 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. as already flushing 2023-04-19 20:53:58,585 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. 2023-04-19 20:53:58,585 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:53:58,585 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:53:58,585 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:58,585 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:53:58,585 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:53:58,589 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:58,589 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:58,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:58,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:53:58,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:53:58,590 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:58,590 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:53:58,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:53:58,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:53:58,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:58,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:58,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:53:58,593 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:53:58,594 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6be4309d[Count = 0] remaining members to acquire global barrier 2023-04-19 20:53:58,594 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:53:58,594 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:58,595 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:58,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:58,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:58,596 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:53:58,596 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:53:58,596 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:53:58,596 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:58,596 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:53:58,597 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:58,597 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:53:58,597 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:53:58,597 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:53:58,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:58,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:53:58,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:53:58,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:53:58,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:53:58,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:58,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:58,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:53:58,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:58,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:58,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:53:58,601 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:53:58,601 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:53:58,601 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:53:58,601 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:53:58,601 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:53:58,601 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:53:58,605 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:58,605 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:58,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:58,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:53:58,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:53:58,606 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:58,606 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:53:58,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:58,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:53:58,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:53:58,607 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681937638581, max timestamp=1681937638605Num uniques 859; 2023-04-19 20:53:58,607 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-19 20:53:58,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:58,608 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681937638581, max timestamp=1681937638605Num uniques 859; 2023-04-19 20:53:58,608 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681937638581, max timestamp=1681937638605Num uniques 859; 2023-04-19 20:53:58,608 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-19 20:53:58,608 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-19 20:53:58,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:53:58,608 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:58,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:58,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:53:58,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:58,610 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:58,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:58,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:53:58,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:58,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:58,614 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:53:58,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:53:58,615 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:53:58,615 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:53:58,615 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:53:58,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:53:58,614 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:53:58,614 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:58,615 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:58,615 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:53:58,615 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:58,615 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:53:58,615 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:53:58,615 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:58,615 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:58,615 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:58,615 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:58,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:53:58,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:53:58,623 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=800 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/C/368c297852e643b79702620e0faf327b 2023-04-19 20:53:58,631 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/A/ccf5eaa3d0f04dc598d3f0b104e1ffdf as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/ccf5eaa3d0f04dc598d3f0b104e1ffdf 2023-04-19 20:53:58,640 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/ccf5eaa3d0f04dc598d3f0b104e1ffdf, entries=150, sequenceid=800, filesize=11.9 K 2023-04-19 20:53:58,641 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/B/15887afc4d5947bd9c14d93ec5e8a98e as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/15887afc4d5947bd9c14d93ec5e8a98e 2023-04-19 20:53:58,647 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/15887afc4d5947bd9c14d93ec5e8a98e, entries=150, sequenceid=800, filesize=11.9 K 2023-04-19 20:53:58,649 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/C/368c297852e643b79702620e0faf327b as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/368c297852e643b79702620e0faf327b 2023-04-19 20:53:58,655 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/368c297852e643b79702620e0faf327b, entries=150, sequenceid=800, filesize=11.9 K 2023-04-19 20:53:58,663 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~66.19 KB/67776, currentSize=87.22 KB/89310 for f70bfe945d0054ce5a228cf255d902f5 in 135ms, sequenceid=800, compaction requested=true 2023-04-19 20:53:58,663 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for f70bfe945d0054ce5a228cf255d902f5: 2023-04-19 20:53:58,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(9158): Flush requested on f70bfe945d0054ce5a228cf255d902f5 2023-04-19 20:53:58,664 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-19 20:53:58,663 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 20:53:58,664 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-19 20:53:58,664 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 20:53:58,664 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 20:53:58,665 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing f70bfe945d0054ce5a228cf255d902f5 3/3 column families, dataSize=93.93 KB heapSize=256.08 KB 2023-04-19 20:53:58,665 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f70bfe945d0054ce5a228cf255d902f5, store=A 2023-04-19 20:53:58,665 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 20:53:58,665 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f70bfe945d0054ce5a228cf255d902f5, store=B 2023-04-19 20:53:58,665 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 20:53:58,665 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f70bfe945d0054ce5a228cf255d902f5, store=C 2023-04-19 20:53:58,666 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 20:53:58,668 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 73764 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-19 20:53:58,668 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1912): f70bfe945d0054ce5a228cf255d902f5/A is initiating minor compaction (all files) 2023-04-19 20:53:58,668 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of f70bfe945d0054ce5a228cf255d902f5/A in TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. 2023-04-19 20:53:58,668 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/b0fe4da425c746a193066d70af799d1a, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/044073d919ed4b08908d57b46c1885e0, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/7f20fbd572a54adc92250879c385c67e, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/9890432f65ab43ec80ca6142349e6f56, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/992f11e8de07463186c620f55d66fbf2, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/ccf5eaa3d0f04dc598d3f0b104e1ffdf] into tmpdir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp, totalSize=72.0 K 2023-04-19 20:53:58,673 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 73764 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-19 20:53:58,673 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting b0fe4da425c746a193066d70af799d1a, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=698, earliestPutTs=1681937633902 2023-04-19 20:53:58,673 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1912): f70bfe945d0054ce5a228cf255d902f5/B is initiating minor compaction (all files) 2023-04-19 20:53:58,673 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2259): Starting compaction of f70bfe945d0054ce5a228cf255d902f5/B in TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. 2023-04-19 20:53:58,673 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/c09cb8f541374249b000d1974a826a55, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/5949adbab09645cfa3cf825e42a09599, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/6f60ac93c4f349848df99b8fa81dc88a, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/62282af0f5144651a6b3d54b7c733b36, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/1b535ff8dc764876bdea11aafd523c70, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/15887afc4d5947bd9c14d93ec5e8a98e] into tmpdir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp, totalSize=72.0 K 2023-04-19 20:53:58,674 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 044073d919ed4b08908d57b46c1885e0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=734, earliestPutTs=1681937634599 2023-04-19 20:53:58,674 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting c09cb8f541374249b000d1974a826a55, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=698, earliestPutTs=1681937633902 2023-04-19 20:53:58,675 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 5949adbab09645cfa3cf825e42a09599, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=734, earliestPutTs=1681937634599 2023-04-19 20:53:58,675 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 7f20fbd572a54adc92250879c385c67e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=747, earliestPutTs=1681937635704 2023-04-19 20:53:58,675 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 6f60ac93c4f349848df99b8fa81dc88a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=747, earliestPutTs=1681937635704 2023-04-19 20:53:58,676 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 9890432f65ab43ec80ca6142349e6f56, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=767, earliestPutTs=1681937636367 2023-04-19 20:53:58,676 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 62282af0f5144651a6b3d54b7c733b36, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=767, earliestPutTs=1681937636367 2023-04-19 20:53:58,677 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 1b535ff8dc764876bdea11aafd523c70, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=779, earliestPutTs=1681937637022 2023-04-19 20:53:58,677 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 15887afc4d5947bd9c14d93ec5e8a98e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=800, earliestPutTs=1681937638203 2023-04-19 20:53:58,678 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 992f11e8de07463186c620f55d66fbf2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=779, earliestPutTs=1681937637022 2023-04-19 20:53:58,679 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting ccf5eaa3d0f04dc598d3f0b104e1ffdf, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=800, earliestPutTs=1681937638203 2023-04-19 20:53:58,694 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] throttle.PressureAwareThroughputController(145): f70bfe945d0054ce5a228cf255d902f5#B#compaction#76 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 20:53:58,698 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] throttle.PressureAwareThroughputController(145): f70bfe945d0054ce5a228cf255d902f5#A#compaction#77 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 20:53:58,700 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=829 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/A/95b8703e92284865bb446131803df1c4 2023-04-19 20:53:58,715 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:53:58,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:53:58,718 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:53:58,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:53:58,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:53:58,725 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:53:58,725 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:53:58,725 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:53:58,725 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:53:58,726 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:58,726 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:58,728 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:53:58,728 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:53:58,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:53:58,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:58,729 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:58,730 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:58,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:53:58,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:58,731 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:53:58,730 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:58,731 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:53:58,732 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/A/5b8f8921d07d4afab18158aee72a417d as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/5b8f8921d07d4afab18158aee72a417d 2023-04-19 20:53:58,736 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:53:58,736 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:53:58,736 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:53:58,736 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:53:58,736 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:53:58,736 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:53:58,736 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:53:58,736 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. 2023-04-19 20:53:58,737 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. started... 2023-04-19 20:53:58,737 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. as already flushing 2023-04-19 20:53:58,737 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. 2023-04-19 20:53:58,737 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:53:58,737 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:53:58,737 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:58,737 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:53:58,737 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:53:58,742 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:58,742 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:58,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:58,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:53:58,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:53:58,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:53:58,742 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:58,742 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:53:58,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:53:58,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:58,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:58,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:53:58,745 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:53:58,745 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3207534d[Count = 0] remaining members to acquire global barrier 2023-04-19 20:53:58,745 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:53:58,745 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:58,746 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in f70bfe945d0054ce5a228cf255d902f5/A of f70bfe945d0054ce5a228cf255d902f5 into 5b8f8921d07d4afab18158aee72a417d(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 20:53:58,746 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for f70bfe945d0054ce5a228cf255d902f5: 2023-04-19 20:53:58,746 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5., storeName=f70bfe945d0054ce5a228cf255d902f5/A, priority=10, startTime=1681937638663; duration=0sec 2023-04-19 20:53:58,746 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 20:53:58,746 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-19 20:53:58,747 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:58,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:58,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:58,747 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:53:58,747 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:53:58,747 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:53:58,748 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:58,748 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:58,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 334 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42350 deadline: 1681937698745, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:58,748 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:53:58,749 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 73764 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-19 20:53:58,749 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1912): f70bfe945d0054ce5a228cf255d902f5/C is initiating minor compaction (all files) 2023-04-19 20:53:58,749 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of f70bfe945d0054ce5a228cf255d902f5/C in TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. 2023-04-19 20:53:58,749 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/9407d1254a5c43de9a87022d7daf7a22, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/fabfaf48242b46048e1d5ab2cd87c458, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/182d5ab2d07a4cae817fe22555780f69, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/6faed4d004564a5da0f1773f2b8fc1b6, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/3b3520a3aa9a4571b8953d385143b0a6, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/368c297852e643b79702620e0faf327b] into tmpdir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp, totalSize=72.0 K 2023-04-19 20:53:58,750 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:58,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:58,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:53:58,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:53:58,750 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:53:58,750 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:53:58,750 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:53:58,750 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 9407d1254a5c43de9a87022d7daf7a22, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=698, earliestPutTs=1681937633902 2023-04-19 20:53:58,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:53:58,751 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting fabfaf48242b46048e1d5ab2cd87c458, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=734, earliestPutTs=1681937634599 2023-04-19 20:53:58,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:53:58,751 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 182d5ab2d07a4cae817fe22555780f69, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=747, earliestPutTs=1681937635704 2023-04-19 20:53:58,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:58,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:58,752 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 6faed4d004564a5da0f1773f2b8fc1b6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=767, earliestPutTs=1681937636367 2023-04-19 20:53:58,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:53:58,752 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 3b3520a3aa9a4571b8953d385143b0a6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=779, earliestPutTs=1681937637022 2023-04-19 20:53:58,753 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 368c297852e643b79702620e0faf327b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=800, earliestPutTs=1681937638203 2023-04-19 20:53:58,753 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:58,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:58,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 255 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42380 deadline: 1681937698746, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:58,754 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:58,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 260 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42342 deadline: 1681937698747, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:58,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:58,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:53:58,756 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:53:58,756 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:53:58,756 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:53:58,756 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:53:58,756 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:53:58,756 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:53:58,758 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:58,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 244 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42366 deadline: 1681937698750, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:58,760 DEBUG [Thread-149] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4156 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5., hostname=jenkins-hbase4.apache.org,35247,1681937616971, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-19 20:53:58,760 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:58,761 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:53:58,760 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:58,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:58,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:58,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:58,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:53:58,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:53:58,761 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:58,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:53:58,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:53:58,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:58,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:53:58,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:58,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:53:58,765 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:58,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:58,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:58,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:53:58,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:58,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:58,769 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:53:58,769 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:53:58,769 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:53:58,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:53:58,770 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:53:58,770 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:53:58,770 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:53:58,770 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:58,770 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:58,770 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:58,770 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:58,770 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:58,770 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:58,770 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:58,770 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:53:58,770 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:53:58,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:53:58,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:53:58,770 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:53:58,778 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] throttle.PressureAwareThroughputController(145): f70bfe945d0054ce5a228cf255d902f5#C#compaction#79 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 20:53:58,853 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:58,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 336 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42350 deadline: 1681937698849, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:58,856 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:58,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 257 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42380 deadline: 1681937698855, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:58,857 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:58,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 262 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42342 deadline: 1681937698855, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:58,870 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:53:58,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:53:58,873 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:53:58,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:53:58,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:53:58,874 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:53:58,874 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:53:58,874 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:53:58,874 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:53:58,875 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:58,875 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:58,877 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:53:58,877 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:58,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:53:58,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:53:58,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:58,878 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:58,878 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:53:58,879 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:58,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:53:58,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:58,881 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:53:58,881 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:53:58,881 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:53:58,881 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:53:58,881 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:53:58,881 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:53:58,881 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:53:58,881 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:53:58,881 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. 2023-04-19 20:53:58,881 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. started... 2023-04-19 20:53:58,881 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. as already flushing 2023-04-19 20:53:58,881 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. 2023-04-19 20:53:58,882 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:53:58,882 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:53:58,882 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:58,882 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:53:58,882 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:53:58,885 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:58,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:58,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:53:58,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:53:58,887 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:58,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:53:58,887 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:58,887 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:53:58,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:53:58,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:58,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:58,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:53:58,889 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:53:58,889 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@72fc98ec[Count = 0] remaining members to acquire global barrier 2023-04-19 20:53:58,889 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:53:58,889 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:58,890 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:58,890 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:58,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:58,891 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:53:58,891 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:53:58,891 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:53:58,891 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:58,891 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:53:58,892 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:58,892 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:53:58,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:58,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:53:58,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:53:58,892 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:53:58,892 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:53:58,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:53:58,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:53:58,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:58,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:58,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:53:58,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:58,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:58,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:53:58,897 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:53:58,897 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:53:58,897 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:53:58,897 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:53:58,897 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:53:58,897 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:53:58,900 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:58,900 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:58,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:58,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:58,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:53:58,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:53:58,900 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:53:58,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:58,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:53:58,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:53:58,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:53:58,900 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:58,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:58,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:58,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:53:58,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:58,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:58,903 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:58,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:53:58,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:58,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:58,906 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:58,906 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:53:58,906 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:58,906 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:53:58,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:53:58,906 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:53:58,906 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:58,906 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:53:58,906 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:53:58,906 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:53:58,906 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:58,907 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:58,907 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:58,907 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:58,907 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:53:58,907 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:53:58,907 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:53:58,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:53:59,007 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:53:59,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:53:59,010 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:53:59,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:53:59,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:53:59,011 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:53:59,011 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:53:59,011 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:53:59,011 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:53:59,013 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:59,013 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:59,014 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:53:59,014 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:59,014 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:53:59,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:53:59,015 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:59,015 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:53:59,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:59,016 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:59,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:53:59,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:59,016 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:53:59,016 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:53:59,017 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:53:59,017 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:53:59,017 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:53:59,017 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:53:59,017 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:53:59,017 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:53:59,018 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. 2023-04-19 20:53:59,018 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. started... 2023-04-19 20:53:59,018 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. as already flushing 2023-04-19 20:53:59,018 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. 2023-04-19 20:53:59,018 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:53:59,018 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:53:59,018 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:59,018 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:53:59,018 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:53:59,019 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:59,019 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:59,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:59,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:53:59,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:53:59,020 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:59,020 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:53:59,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:53:59,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:53:59,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:59,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:59,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:53:59,023 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:53:59,023 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@79e3054c[Count = 0] remaining members to acquire global barrier 2023-04-19 20:53:59,023 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:53:59,023 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:59,027 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:59,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:59,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:59,028 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:53:59,028 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:53:59,028 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:59,028 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:53:59,028 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:53:59,029 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:53:59,029 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:53:59,029 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:53:59,029 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:59,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:59,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:53:59,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:53:59,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:53:59,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:53:59,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:59,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:59,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:53:59,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:59,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:59,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:53:59,033 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:53:59,033 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:53:59,033 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:53:59,033 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:53:59,033 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:53:59,033 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:53:59,035 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:59,035 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:59,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:59,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:59,035 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:53:59,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:59,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:53:59,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:53:59,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:53:59,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:53:59,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:53:59,036 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:59,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:59,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:59,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:53:59,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:59,037 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:59,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:59,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:53:59,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:59,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:59,039 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:53:59,039 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:53:59,039 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:53:59,039 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:53:59,039 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:59,039 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:53:59,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:53:59,040 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:53:59,040 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:53:59,039 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:53:59,040 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:53:59,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:53:59,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:53:59,039 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:59,040 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:59,040 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:59,040 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:59,040 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:59,040 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:59,058 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:59,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 338 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42350 deadline: 1681937699055, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:59,060 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:59,060 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:59,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 264 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42342 deadline: 1681937699059, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:59,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 259 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42380 deadline: 1681937699058, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:59,111 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/B/fa7ebf4413e5438d84e156608c64a1ec as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/fa7ebf4413e5438d84e156608c64a1ec 2023-04-19 20:53:59,119 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in f70bfe945d0054ce5a228cf255d902f5/B of f70bfe945d0054ce5a228cf255d902f5 into fa7ebf4413e5438d84e156608c64a1ec(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 20:53:59,119 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for f70bfe945d0054ce5a228cf255d902f5: 2023-04-19 20:53:59,119 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5., storeName=f70bfe945d0054ce5a228cf255d902f5/B, priority=10, startTime=1681937638664; duration=0sec 2023-04-19 20:53:59,120 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 20:53:59,120 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=829 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/B/bdc9fcc38d60479498d1a84e4de8f269 2023-04-19 20:53:59,140 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:53:59,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:53:59,142 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:53:59,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:53:59,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:53:59,144 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:53:59,144 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:53:59,144 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:53:59,144 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:53:59,144 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:59,144 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:59,150 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:53:59,150 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:59,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:53:59,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:53:59,150 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:59,151 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:53:59,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:59,151 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:59,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:53:59,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:59,152 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:53:59,152 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:53:59,152 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:53:59,152 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:53:59,152 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:53:59,152 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:53:59,152 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:53:59,152 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:53:59,152 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. 2023-04-19 20:53:59,152 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. started... 2023-04-19 20:53:59,153 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. as already flushing 2023-04-19 20:53:59,153 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. 2023-04-19 20:53:59,153 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:53:59,153 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:53:59,153 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:59,153 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:53:59,153 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:53:59,155 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:59,155 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:59,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:59,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:53:59,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:53:59,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:53:59,156 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:59,156 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:53:59,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:53:59,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:59,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:59,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:53:59,157 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:53:59,157 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@33c9b235[Count = 0] remaining members to acquire global barrier 2023-04-19 20:53:59,157 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:53:59,157 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:59,159 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:59,159 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:59,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:59,159 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:53:59,159 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:53:59,159 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:53:59,159 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:59,159 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:53:59,161 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:59,161 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:53:59,161 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:53:59,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:59,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:53:59,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:53:59,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:53:59,161 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:53:59,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:53:59,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:59,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:59,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:53:59,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:59,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:59,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:53:59,164 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:53:59,164 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:53:59,165 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:53:59,165 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:53:59,165 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:53:59,165 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:53:59,168 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:59,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:59,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:53:59,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:53:59,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:53:59,169 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:59,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:59,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:53:59,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:59,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:59,170 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:59,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:53:59,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:59,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:59,174 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:59,174 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:59,174 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:59,174 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:59,174 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:59,174 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:59,174 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:59,175 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:53:59,175 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:53:59,175 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:53:59,175 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:53:59,175 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:53:59,176 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:59,176 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:53:59,176 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:59,176 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:53:59,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:59,178 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:53:59,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:53:59,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:53:59,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:53:59,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:53:59,191 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/C/4a626fd6bf424606b77ae02cf5c7697f as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/4a626fd6bf424606b77ae02cf5c7697f 2023-04-19 20:53:59,197 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in f70bfe945d0054ce5a228cf255d902f5/C of f70bfe945d0054ce5a228cf255d902f5 into 4a626fd6bf424606b77ae02cf5c7697f(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 20:53:59,197 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for f70bfe945d0054ce5a228cf255d902f5: 2023-04-19 20:53:59,197 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5., storeName=f70bfe945d0054ce5a228cf255d902f5/C, priority=10, startTime=1681937638664; duration=0sec 2023-04-19 20:53:59,198 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 20:53:59,275 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:53:59,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:53:59,278 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:53:59,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:53:59,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:53:59,287 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:53:59,287 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:53:59,288 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:53:59,288 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:53:59,289 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:59,289 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:59,292 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:53:59,292 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:59,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:53:59,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:53:59,292 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:59,292 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:53:59,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:59,293 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:59,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:53:59,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:59,294 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:53:59,296 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:53:59,296 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:53:59,296 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:53:59,296 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:53:59,296 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:53:59,297 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:53:59,297 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:53:59,297 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. 2023-04-19 20:53:59,297 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. started... 2023-04-19 20:53:59,297 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. as already flushing 2023-04-19 20:53:59,297 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. 2023-04-19 20:53:59,297 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:53:59,297 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:53:59,297 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:59,297 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:53:59,297 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:53:59,299 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:59,300 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:59,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:59,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:53:59,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:53:59,300 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:59,300 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:53:59,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:53:59,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:53:59,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:59,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:59,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:53:59,309 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:53:59,309 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3b5c123a[Count = 0] remaining members to acquire global barrier 2023-04-19 20:53:59,309 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:53:59,309 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:59,313 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:59,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:59,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:59,313 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:53:59,313 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:53:59,313 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:53:59,315 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:59,315 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:53:59,317 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:53:59,317 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:53:59,317 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:53:59,320 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:59,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:59,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:53:59,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:53:59,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:53:59,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:53:59,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:59,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:59,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:53:59,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:59,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:59,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:53:59,322 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:53:59,322 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:53:59,322 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:53:59,323 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:53:59,323 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:53:59,323 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:53:59,326 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:59,326 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:59,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:59,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:59,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:59,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:53:59,327 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:53:59,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:53:59,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:53:59,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:53:59,328 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:59,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:53:59,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:59,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:59,330 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:59,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:53:59,334 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:59,334 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:59,334 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:59,334 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:59,334 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:53:59,334 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:53:59,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:53:59,334 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:53:59,334 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:53:59,334 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:53:59,334 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:59,334 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:59,334 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:59,334 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:53:59,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:53:59,334 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:53:59,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:53:59,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:53:59,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:53:59,337 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:53:59,363 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:59,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 340 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42350 deadline: 1681937699361, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:59,363 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:59,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 266 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42342 deadline: 1681937699361, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:59,364 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:59,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 261 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42380 deadline: 1681937699362, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:59,434 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:53:59,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:53:59,437 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:53:59,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:53:59,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:53:59,438 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:53:59,438 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:53:59,438 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:53:59,438 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:53:59,439 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:59,439 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:59,441 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:53:59,441 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:59,441 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:53:59,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:53:59,442 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:59,442 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:53:59,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:59,442 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:59,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:53:59,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:59,443 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:53:59,443 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:53:59,443 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:53:59,443 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:53:59,443 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:53:59,443 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:53:59,443 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:53:59,443 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:53:59,443 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. 2023-04-19 20:53:59,443 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. started... 2023-04-19 20:53:59,444 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. as already flushing 2023-04-19 20:53:59,444 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. 2023-04-19 20:53:59,444 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:53:59,444 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:53:59,444 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:59,444 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:53:59,444 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:53:59,445 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:59,445 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:59,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:59,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:53:59,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:53:59,446 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:59,446 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:53:59,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:53:59,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:53:59,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:59,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:59,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:53:59,448 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:53:59,448 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2df1d645[Count = 0] remaining members to acquire global barrier 2023-04-19 20:53:59,448 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:53:59,448 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:59,449 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:59,450 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:59,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:59,450 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:53:59,450 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:59,450 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:53:59,450 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:53:59,450 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:53:59,454 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:59,454 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:53:59,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:59,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:53:59,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:53:59,455 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:53:59,455 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:53:59,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:53:59,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:53:59,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:59,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:59,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:53:59,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:59,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:59,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:53:59,458 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:53:59,458 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:53:59,458 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:53:59,458 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:53:59,458 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:53:59,458 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:53:59,459 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:59,459 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:59,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:59,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:59,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:53:59,460 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:53:59,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:53:59,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:59,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:53:59,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:53:59,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:53:59,461 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:59,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:59,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:59,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:53:59,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:59,468 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:59,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:59,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:53:59,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:53:59,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:59,471 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:59,471 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:53:59,471 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:59,471 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:53:59,471 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:53:59,471 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:53:59,471 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:53:59,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:53:59,471 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:53:59,471 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:53:59,471 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:59,471 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:53:59,471 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:59,471 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:53:59,471 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:59,471 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:53:59,471 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:59,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:53:59,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:53:59,540 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=829 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/C/706dc5a4d58e4ef790b9a621091cf852 2023-04-19 20:53:59,546 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/A/95b8703e92284865bb446131803df1c4 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/95b8703e92284865bb446131803df1c4 2023-04-19 20:53:59,554 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/95b8703e92284865bb446131803df1c4, entries=200, sequenceid=829, filesize=14.3 K 2023-04-19 20:53:59,555 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/B/bdc9fcc38d60479498d1a84e4de8f269 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/bdc9fcc38d60479498d1a84e4de8f269 2023-04-19 20:53:59,561 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/bdc9fcc38d60479498d1a84e4de8f269, entries=150, sequenceid=829, filesize=11.9 K 2023-04-19 20:53:59,562 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/C/706dc5a4d58e4ef790b9a621091cf852 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/706dc5a4d58e4ef790b9a621091cf852 2023-04-19 20:53:59,568 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/706dc5a4d58e4ef790b9a621091cf852, entries=150, sequenceid=829, filesize=11.9 K 2023-04-19 20:53:59,570 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~256.03 KB/262176, currentSize=93.93 KB/96180 for f70bfe945d0054ce5a228cf255d902f5 in 905ms, sequenceid=829, compaction requested=false 2023-04-19 20:53:59,570 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for f70bfe945d0054ce5a228cf255d902f5: 2023-04-19 20:53:59,571 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:53:59,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:53:59,574 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:53:59,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:53:59,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:53:59,576 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:53:59,576 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:53:59,576 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:53:59,576 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:53:59,576 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:59,576 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:59,578 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:53:59,578 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:59,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:53:59,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:53:59,578 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:59,578 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:53:59,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:59,579 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:53:59,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:53:59,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:53:59,579 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:53:59,580 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:53:59,580 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:53:59,580 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:53:59,580 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:53:59,580 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:53:59,584 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:53:59,584 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:53:59,584 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. 2023-04-19 20:53:59,584 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. started... 2023-04-19 20:53:59,584 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing f70bfe945d0054ce5a228cf255d902f5 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-19 20:53:59,584 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f70bfe945d0054ce5a228cf255d902f5, store=A 2023-04-19 20:53:59,584 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:53:59,585 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f70bfe945d0054ce5a228cf255d902f5, store=B 2023-04-19 20:53:59,585 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:53:59,585 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f70bfe945d0054ce5a228cf255d902f5, store=C 2023-04-19 20:53:59,585 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:53:59,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(9158): Flush requested on f70bfe945d0054ce5a228cf255d902f5 2023-04-19 20:53:59,866 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. as already flushing 2023-04-19 20:53:59,877 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:59,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 345 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42350 deadline: 1681937699875, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:59,878 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:59,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 271 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42342 deadline: 1681937699876, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:59,879 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:59,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 269 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42380 deadline: 1681937699877, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:59,979 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:59,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 347 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42350 deadline: 1681937699978, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:59,980 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:59,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 273 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42342 deadline: 1681937699979, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:59,982 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:53:59,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 271 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42380 deadline: 1681937699980, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:53:59,997 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=849 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/A/3ee37a4c3c004b9cb346b73b5d6177b3 2023-04-19 20:54:00,010 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=849 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/B/d9e6eca8675c4dabb234ea1489df690c 2023-04-19 20:54:00,182 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:00,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 349 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42350 deadline: 1681937700180, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:00,183 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:00,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 275 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42342 deadline: 1681937700182, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:00,184 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:00,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 273 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42380 deadline: 1681937700183, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:00,427 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=849 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/C/ea802b8e8b0e4d2e836df64a749947ee 2023-04-19 20:54:00,432 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/A/3ee37a4c3c004b9cb346b73b5d6177b3 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/3ee37a4c3c004b9cb346b73b5d6177b3 2023-04-19 20:54:00,438 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/3ee37a4c3c004b9cb346b73b5d6177b3, entries=150, sequenceid=849, filesize=11.9 K 2023-04-19 20:54:00,439 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/B/d9e6eca8675c4dabb234ea1489df690c as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/d9e6eca8675c4dabb234ea1489df690c 2023-04-19 20:54:00,449 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/d9e6eca8675c4dabb234ea1489df690c, entries=150, sequenceid=849, filesize=11.9 K 2023-04-19 20:54:00,450 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/C/ea802b8e8b0e4d2e836df64a749947ee as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/ea802b8e8b0e4d2e836df64a749947ee 2023-04-19 20:54:00,454 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/ea802b8e8b0e4d2e836df64a749947ee, entries=150, sequenceid=849, filesize=11.9 K 2023-04-19 20:54:00,455 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for f70bfe945d0054ce5a228cf255d902f5 in 871ms, sequenceid=849, compaction requested=true 2023-04-19 20:54:00,455 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f70bfe945d0054ce5a228cf255d902f5: 2023-04-19 20:54:00,455 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. 2023-04-19 20:54:00,455 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:54:00,455 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:54:00,455 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:00,455 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:54:00,456 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:54:00,458 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:00,458 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:00,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:00,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:00,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:00,458 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:00,458 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:54:00,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:00,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:00,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:00,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:00,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:00,460 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:54:00,460 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4c596fd2[Count = 0] remaining members to acquire global barrier 2023-04-19 20:54:00,460 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:54:00,460 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:00,461 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:00,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:00,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:00,461 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:54:00,461 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:54:00,461 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:54:00,463 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:00,463 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:54:00,464 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:54:00,464 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:00,464 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:00,464 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:54:00,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:00,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:00,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:00,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:00,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:00,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:00,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:00,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:00,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:00,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:00,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:54:00,467 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:54:00,467 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:54:00,467 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:54:00,467 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:54:00,467 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:54:00,467 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:54:00,468 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:00,468 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:00,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:00,469 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:00,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:00,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:00,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:00,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:00,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:00,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:00,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:00,469 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:00,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:00,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:00,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:00,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:00,470 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:00,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:00,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:00,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:00,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:00,475 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:00,475 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:00,475 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:00,475 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:00,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:00,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:00,475 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:54:00,475 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:54:00,476 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:54:00,476 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:54:00,475 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:00,475 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:00,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:00,476 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:54:00,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:00,476 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:00,476 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:00,476 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:00,476 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:00,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(9158): Flush requested on f70bfe945d0054ce5a228cf255d902f5 2023-04-19 20:54:00,487 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing f70bfe945d0054ce5a228cf255d902f5 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-19 20:54:00,488 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f70bfe945d0054ce5a228cf255d902f5, store=A 2023-04-19 20:54:00,488 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:54:00,488 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f70bfe945d0054ce5a228cf255d902f5, store=B 2023-04-19 20:54:00,488 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:54:00,488 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f70bfe945d0054ce5a228cf255d902f5, store=C 2023-04-19 20:54:00,488 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:54:00,502 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:00,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 282 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42342 deadline: 1681937700501, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:00,505 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:00,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 355 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42350 deadline: 1681937700504, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:00,506 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:00,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 280 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42380 deadline: 1681937700504, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:00,521 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=866 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/A/4fda88ae9c5b46c0a5f24c6c1cba1231 2023-04-19 20:54:00,576 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:54:00,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:54:00,579 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:54:00,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:54:00,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:54:00,583 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:54:00,583 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:00,583 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:54:00,583 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:54:00,584 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:00,584 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:00,585 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:00,585 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:00,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:00,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:00,585 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:00,585 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:54:00,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:00,586 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:00,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:54:00,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:00,586 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:54:00,592 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:54:00,592 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:54:00,592 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:54:00,592 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:00,592 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:54:00,593 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:54:00,593 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:54:00,593 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. 2023-04-19 20:54:00,593 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. started... 2023-04-19 20:54:00,593 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. as already flushing 2023-04-19 20:54:00,593 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. 2023-04-19 20:54:00,593 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:54:00,593 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:54:00,593 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:00,593 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:54:00,593 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:54:00,595 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:00,595 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:00,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:00,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:00,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:00,595 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:00,595 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:54:00,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:00,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:00,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:00,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:00,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:00,597 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:54:00,597 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@327f3476[Count = 0] remaining members to acquire global barrier 2023-04-19 20:54:00,597 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:54:00,597 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:00,598 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:00,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:00,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:00,598 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:00,598 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:54:00,598 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:54:00,598 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:54:00,598 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:54:00,602 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:54:00,602 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:00,602 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:00,602 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:54:00,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:00,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:00,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:00,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:00,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:00,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:00,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:00,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:00,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:00,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:00,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:54:00,605 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:54:00,605 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:54:00,605 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:54:00,605 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:54:00,605 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:54:00,605 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:54:00,606 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:00,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 284 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42342 deadline: 1681937700604, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:00,607 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:00,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:00,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:00,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:00,607 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:00,607 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:00,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:00,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:00,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:00,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:00,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:00,608 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:00,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:00,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:00,608 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:00,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 357 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42350 deadline: 1681937700607, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:00,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:00,609 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:00,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 282 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42380 deadline: 1681937700607, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:00,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:00,610 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:00,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:00,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:00,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:00,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:00,613 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:00,613 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:00,613 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:00,613 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:00,613 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:00,613 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:00,613 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:54:00,613 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:54:00,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:00,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:00,613 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:00,613 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:00,613 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:54:00,613 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:00,613 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:00,613 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:54:00,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:00,614 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:54:00,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:00,714 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:54:00,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:54:00,716 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:54:00,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:54:00,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:54:00,717 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:54:00,717 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:00,717 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:54:00,717 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:54:00,718 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:00,718 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:00,719 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:00,719 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:00,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:00,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:00,720 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:00,720 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:54:00,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:00,720 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:00,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:54:00,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:00,720 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:54:00,721 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:54:00,721 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:54:00,721 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:54:00,721 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:00,721 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:54:00,721 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:54:00,721 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:54:00,721 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. 2023-04-19 20:54:00,721 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. started... 2023-04-19 20:54:00,721 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. as already flushing 2023-04-19 20:54:00,721 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. 2023-04-19 20:54:00,721 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:54:00,721 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:54:00,721 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:00,721 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:54:00,721 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:54:00,724 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:00,724 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:00,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:00,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:00,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:00,724 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:00,724 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:54:00,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:00,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:00,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:00,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:00,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:00,729 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:54:00,729 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@69692dc1[Count = 0] remaining members to acquire global barrier 2023-04-19 20:54:00,729 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:54:00,729 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:00,731 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:00,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:00,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:00,731 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:54:00,731 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:54:00,731 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:54:00,731 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:00,731 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:54:00,732 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:00,732 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:54:00,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:00,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:00,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:00,732 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:00,733 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:54:00,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:00,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:00,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:00,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:00,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:00,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:00,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:00,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:54:00,735 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:54:00,735 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:54:00,735 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:54:00,735 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:54:00,735 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:54:00,735 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:54:00,736 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:00,736 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:00,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:00,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:00,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:00,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:00,736 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:00,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:00,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:00,737 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:00,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:00,737 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:00,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:00,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:00,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:00,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:00,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:00,738 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:00,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:00,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:00,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:00,744 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:00,744 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:00,744 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:00,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:00,744 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:00,744 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:54:00,744 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:54:00,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:00,744 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:00,744 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:00,744 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:00,744 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:00,745 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:00,745 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:54:00,745 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:00,745 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:54:00,745 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:00,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:00,808 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:00,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 286 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42342 deadline: 1681937700808, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:00,810 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:00,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 359 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42350 deadline: 1681937700809, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:00,811 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:00,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 284 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42380 deadline: 1681937700811, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:00,845 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:54:00,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:54:00,847 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:54:00,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:54:00,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:54:00,848 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:54:00,848 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:00,848 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:54:00,848 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:54:00,848 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:00,848 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:00,850 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:00,850 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:00,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:00,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:00,850 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:00,850 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:54:00,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:00,850 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:00,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:54:00,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:00,851 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:54:00,851 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:54:00,851 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:54:00,851 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:54:00,851 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:00,851 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:54:00,851 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:54:00,851 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:54:00,852 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. 2023-04-19 20:54:00,852 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. started... 2023-04-19 20:54:00,852 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. as already flushing 2023-04-19 20:54:00,852 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. 2023-04-19 20:54:00,852 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:54:00,852 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:54:00,852 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:00,852 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:54:00,852 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:54:00,854 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:00,854 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:00,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:00,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:00,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:00,854 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:00,854 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:54:00,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:00,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:00,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:00,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:00,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:00,856 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:54:00,856 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4a099d57[Count = 0] remaining members to acquire global barrier 2023-04-19 20:54:00,856 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:54:00,856 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:00,858 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:00,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:00,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:00,859 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:54:00,859 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:54:00,859 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:54:00,859 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:00,859 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:54:00,861 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:00,861 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:54:00,861 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:00,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:00,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:00,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:00,861 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:54:00,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:00,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:00,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:00,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:00,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:00,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:00,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:00,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:54:00,863 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:54:00,863 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:54:00,863 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:54:00,863 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:54:00,863 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:54:00,863 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:54:00,865 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:00,865 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:00,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:00,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:00,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:00,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:00,865 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:00,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:00,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:00,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:00,866 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:00,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:00,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:00,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:00,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:00,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:00,868 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:00,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:00,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:00,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:00,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:00,871 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:00,871 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:00,871 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:00,871 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:54:00,871 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:54:00,871 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:00,871 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:00,871 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:00,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:00,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:00,872 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:00,872 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:00,872 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:00,872 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:00,872 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:54:00,872 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:54:00,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:00,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:00,951 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=866 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/B/8b66d3b9abca4e21872bee111873d51a 2023-04-19 20:54:00,972 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:54:00,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:54:00,974 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:54:00,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:54:00,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:54:00,975 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:54:00,975 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:00,975 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:54:00,975 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:54:00,976 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:00,976 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:00,977 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:00,977 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:00,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:00,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:00,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:00,978 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:00,978 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:54:00,978 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:00,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:54:00,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:00,979 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:54:00,979 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:54:00,979 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:54:00,979 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:54:00,979 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:00,979 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:54:00,980 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:54:00,980 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:54:00,980 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. 2023-04-19 20:54:00,980 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. started... 2023-04-19 20:54:00,980 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. as already flushing 2023-04-19 20:54:00,980 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. 2023-04-19 20:54:00,980 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:54:00,980 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:54:00,980 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:00,980 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:54:00,980 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:54:00,982 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:00,982 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:00,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:00,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:00,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:00,983 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:00,983 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:54:00,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:00,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:00,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:00,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:00,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:00,984 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:54:00,984 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d294c3b[Count = 0] remaining members to acquire global barrier 2023-04-19 20:54:00,984 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:54:00,984 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:00,985 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:00,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:00,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:00,986 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:54:00,986 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:54:00,986 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:54:00,986 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:00,986 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:54:00,987 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:00,987 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:54:00,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:00,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:00,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:00,987 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:00,988 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:54:00,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:00,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:00,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:00,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:00,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:00,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:00,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:00,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:54:00,990 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:54:00,990 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:54:00,990 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:54:00,990 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:54:00,990 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:54:00,990 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:54:00,992 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:00,992 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:00,992 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:00,992 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:00,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:00,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:00,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:00,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:00,992 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:00,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:00,992 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:00,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:00,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:00,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:00,994 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:00,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:00,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:00,997 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:00,997 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:00,997 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:00,997 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:00,997 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:00,997 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:00,997 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:54:00,997 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:54:00,997 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:00,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:00,997 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:00,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:00,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:00,997 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:00,998 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:00,998 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:54:00,998 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:54:00,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:00,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:00,999 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:54:01,098 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:54:01,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:54:01,101 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:54:01,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:54:01,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:54:01,102 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:54:01,102 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:01,102 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:54:01,102 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:54:01,102 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:01,102 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:01,104 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:01,104 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:01,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:01,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:01,104 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:01,104 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:54:01,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:01,105 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:01,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:54:01,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:01,105 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:54:01,105 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:54:01,105 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:54:01,106 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:54:01,106 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:01,106 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:54:01,106 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:54:01,106 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:54:01,106 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. 2023-04-19 20:54:01,106 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. started... 2023-04-19 20:54:01,106 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. as already flushing 2023-04-19 20:54:01,106 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. 2023-04-19 20:54:01,106 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:54:01,106 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:54:01,106 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:01,106 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:54:01,106 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:54:01,108 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:01,108 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:01,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:01,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:01,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:01,109 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:01,109 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:54:01,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:01,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:01,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:01,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:01,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:01,111 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:54:01,111 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6be1d6c[Count = 0] remaining members to acquire global barrier 2023-04-19 20:54:01,111 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:54:01,111 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:01,112 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:01,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:01,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:01,112 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:54:01,112 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:54:01,112 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:01,112 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:01,112 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:54:01,112 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:54:01,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 288 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42342 deadline: 1681937701110, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:01,113 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:01,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 361 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42350 deadline: 1681937701111, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:01,114 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:01,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 286 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42380 deadline: 1681937701114, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:01,117 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:01,117 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:54:01,117 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:01,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:01,117 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:54:01,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:01,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:01,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:01,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:01,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:01,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:01,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:01,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:01,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:01,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:54:01,121 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:54:01,121 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:54:01,121 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:54:01,121 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:54:01,121 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:54:01,121 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:54:01,122 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:01,122 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:01,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:01,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:01,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:01,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:01,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:01,123 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:01,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:01,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:01,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:01,124 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:01,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:01,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:01,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:01,126 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:01,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:01,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:01,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:01,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:01,131 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:01,132 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:01,132 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:01,132 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:01,132 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:01,132 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:01,132 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:01,132 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:01,132 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:01,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:01,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:01,132 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:01,132 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:54:01,132 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:54:01,132 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:54:01,133 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:54:01,136 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:01,136 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:54:01,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:01,233 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:54:01,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:54:01,235 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:54:01,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:54:01,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:54:01,236 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:54:01,236 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:01,236 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:54:01,236 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:54:01,237 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:01,237 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:01,242 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:01,242 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:01,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:01,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:01,242 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:01,242 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:54:01,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:01,242 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:01,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:54:01,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:01,243 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:54:01,243 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:54:01,243 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:54:01,243 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:54:01,243 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:01,243 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:54:01,243 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:54:01,243 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:54:01,243 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. 2023-04-19 20:54:01,244 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. started... 2023-04-19 20:54:01,244 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. as already flushing 2023-04-19 20:54:01,244 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. 2023-04-19 20:54:01,244 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:54:01,244 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:54:01,244 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:01,244 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:54:01,244 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:54:01,245 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:01,245 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:01,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:01,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:01,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:01,246 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:01,246 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:54:01,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:01,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:01,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:01,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:01,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:01,248 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:54:01,248 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@26993f5f[Count = 0] remaining members to acquire global barrier 2023-04-19 20:54:01,248 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:54:01,248 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:01,249 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:01,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:01,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:01,249 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:54:01,249 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:54:01,249 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:54:01,249 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:01,249 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:54:01,251 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:01,251 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:54:01,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:01,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:01,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:01,251 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:01,251 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:54:01,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:01,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:01,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:01,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:01,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:01,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:01,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:01,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:54:01,254 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:54:01,254 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:54:01,254 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:54:01,254 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:54:01,254 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:54:01,254 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:54:01,256 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:01,256 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:01,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:01,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:01,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:01,257 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:01,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:01,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:01,257 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:01,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:01,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:01,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:01,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:01,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:01,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:01,258 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:01,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:01,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:01,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:01,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:01,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:01,260 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:01,260 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:01,260 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:01,260 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:01,260 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:01,260 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:01,260 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:54:01,260 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:54:01,260 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:01,260 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:01,261 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:54:01,261 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:54:01,261 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:54:01,264 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:01,264 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:01,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:01,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:01,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:01,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:01,361 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:54:01,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:54:01,363 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:54:01,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:54:01,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:54:01,364 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:54:01,364 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:01,364 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:54:01,364 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:54:01,365 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:01,365 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:01,365 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=866 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/C/09b23b0533b94d2b99a56701c9476800 2023-04-19 20:54:01,367 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:01,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:01,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:01,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:01,368 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:01,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:54:01,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:01,370 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:54:01,370 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:54:01,370 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:54:01,370 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:54:01,370 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:01,370 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:54:01,371 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/A/4fda88ae9c5b46c0a5f24c6c1cba1231 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/4fda88ae9c5b46c0a5f24c6c1cba1231 2023-04-19 20:54:01,372 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. 2023-04-19 20:54:01,372 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. started... 2023-04-19 20:54:01,371 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:54:01,372 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:54:01,372 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. as already flushing 2023-04-19 20:54:01,372 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. 2023-04-19 20:54:01,372 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:54:01,372 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:54:01,372 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:01,372 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:54:01,372 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:54:01,374 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:01,374 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:01,374 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:54:01,377 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/4fda88ae9c5b46c0a5f24c6c1cba1231, entries=250, sequenceid=866, filesize=16.7 K 2023-04-19 20:54:01,378 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/B/8b66d3b9abca4e21872bee111873d51a as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/8b66d3b9abca4e21872bee111873d51a 2023-04-19 20:54:01,379 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:01,380 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:01,380 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:54:01,380 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a1fffda[Count = 0] remaining members to acquire global barrier 2023-04-19 20:54:01,380 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:54:01,380 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:54:01,380 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:01,382 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:01,382 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:01,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:01,383 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:01,383 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:54:01,383 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:54:01,383 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:54:01,383 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:54:01,384 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/8b66d3b9abca4e21872bee111873d51a, entries=150, sequenceid=866, filesize=11.9 K 2023-04-19 20:54:01,385 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/C/09b23b0533b94d2b99a56701c9476800 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/09b23b0533b94d2b99a56701c9476800 2023-04-19 20:54:01,385 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:01,385 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:54:01,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:01,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:01,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:01,385 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:01,385 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:54:01,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:01,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:01,391 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/09b23b0533b94d2b99a56701c9476800, entries=150, sequenceid=866, filesize=11.9 K 2023-04-19 20:54:01,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:01,392 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for f70bfe945d0054ce5a228cf255d902f5 in 905ms, sequenceid=866, compaction requested=true 2023-04-19 20:54:01,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:01,392 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for f70bfe945d0054ce5a228cf255d902f5: 2023-04-19 20:54:01,392 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 20:54:01,392 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-19 20:54:01,392 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-19 20:54:01,392 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 20:54:01,393 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 20:54:01,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:01,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:01,394 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 56916 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-19 20:54:01,394 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1912): f70bfe945d0054ce5a228cf255d902f5/A is initiating minor compaction (all files) 2023-04-19 20:54:01,394 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of f70bfe945d0054ce5a228cf255d902f5/A in TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. 2023-04-19 20:54:01,394 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/5b8f8921d07d4afab18158aee72a417d, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/95b8703e92284865bb446131803df1c4, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/3ee37a4c3c004b9cb346b73b5d6177b3, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/4fda88ae9c5b46c0a5f24c6c1cba1231] into tmpdir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp, totalSize=55.6 K 2023-04-19 20:54:01,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:01,395 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 5b8f8921d07d4afab18158aee72a417d, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=800, earliestPutTs=1681937638203 2023-04-19 20:54:01,395 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49596 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-19 20:54:01,395 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1912): f70bfe945d0054ce5a228cf255d902f5/B is initiating minor compaction (all files) 2023-04-19 20:54:01,395 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2259): Starting compaction of f70bfe945d0054ce5a228cf255d902f5/B in TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. 2023-04-19 20:54:01,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:54:01,395 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/fa7ebf4413e5438d84e156608c64a1ec, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/bdc9fcc38d60479498d1a84e4de8f269, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/d9e6eca8675c4dabb234ea1489df690c, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/8b66d3b9abca4e21872bee111873d51a] into tmpdir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp, totalSize=48.4 K 2023-04-19 20:54:01,395 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:54:01,395 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:54:01,395 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:54:01,395 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:54:01,395 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:54:01,395 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:54:01,395 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 95b8703e92284865bb446131803df1c4, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=829, earliestPutTs=1681937638614 2023-04-19 20:54:01,396 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting fa7ebf4413e5438d84e156608c64a1ec, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=800, earliestPutTs=1681937638203 2023-04-19 20:54:01,396 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 3ee37a4c3c004b9cb346b73b5d6177b3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=849, earliestPutTs=1681937638728 2023-04-19 20:54:01,396 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting bdc9fcc38d60479498d1a84e4de8f269, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=829, earliestPutTs=1681937638614 2023-04-19 20:54:01,397 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting d9e6eca8675c4dabb234ea1489df690c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=849, earliestPutTs=1681937638728 2023-04-19 20:54:01,397 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 4fda88ae9c5b46c0a5f24c6c1cba1231, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=866, earliestPutTs=1681937639874 2023-04-19 20:54:01,397 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 8b66d3b9abca4e21872bee111873d51a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=866, earliestPutTs=1681937639874 2023-04-19 20:54:01,397 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:01,397 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:01,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:01,398 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:01,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:01,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:01,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:01,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:01,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:01,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:01,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:01,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:01,399 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:01,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:01,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:01,401 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:01,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:01,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:01,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:01,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:01,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:01,404 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:01,404 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:01,404 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:01,404 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:01,405 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:01,405 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:01,405 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:54:01,405 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:54:01,404 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:01,405 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:01,405 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:01,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:01,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:01,405 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:01,405 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:54:01,405 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:54:01,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:01,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:01,406 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:54:01,414 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] throttle.PressureAwareThroughputController(145): f70bfe945d0054ce5a228cf255d902f5#B#compaction#87 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 20:54:01,418 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] throttle.PressureAwareThroughputController(145): f70bfe945d0054ce5a228cf255d902f5#A#compaction#88 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 20:54:01,438 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/A/791cf76d266e4f2d9ccd8b4ad5f0d865 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/791cf76d266e4f2d9ccd8b4ad5f0d865 2023-04-19 20:54:01,447 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in f70bfe945d0054ce5a228cf255d902f5/A of f70bfe945d0054ce5a228cf255d902f5 into 791cf76d266e4f2d9ccd8b4ad5f0d865(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 20:54:01,448 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for f70bfe945d0054ce5a228cf255d902f5: 2023-04-19 20:54:01,448 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5., storeName=f70bfe945d0054ce5a228cf255d902f5/A, priority=12, startTime=1681937641392; duration=0sec 2023-04-19 20:54:01,448 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 20:54:01,448 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-19 20:54:01,450 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49596 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-19 20:54:01,450 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1912): f70bfe945d0054ce5a228cf255d902f5/C is initiating minor compaction (all files) 2023-04-19 20:54:01,450 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of f70bfe945d0054ce5a228cf255d902f5/C in TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. 2023-04-19 20:54:01,450 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/4a626fd6bf424606b77ae02cf5c7697f, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/706dc5a4d58e4ef790b9a621091cf852, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/ea802b8e8b0e4d2e836df64a749947ee, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/09b23b0533b94d2b99a56701c9476800] into tmpdir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp, totalSize=48.4 K 2023-04-19 20:54:01,451 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 4a626fd6bf424606b77ae02cf5c7697f, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=800, earliestPutTs=1681937638203 2023-04-19 20:54:01,451 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 706dc5a4d58e4ef790b9a621091cf852, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=829, earliestPutTs=1681937638614 2023-04-19 20:54:01,452 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting ea802b8e8b0e4d2e836df64a749947ee, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=849, earliestPutTs=1681937638728 2023-04-19 20:54:01,452 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 09b23b0533b94d2b99a56701c9476800, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=866, earliestPutTs=1681937639874 2023-04-19 20:54:01,465 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] throttle.PressureAwareThroughputController(145): f70bfe945d0054ce5a228cf255d902f5#C#compaction#89 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 20:54:01,484 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/C/1f3dde650ce7452fa9d41a5318b8df9f as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/1f3dde650ce7452fa9d41a5318b8df9f 2023-04-19 20:54:01,493 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in f70bfe945d0054ce5a228cf255d902f5/C of f70bfe945d0054ce5a228cf255d902f5 into 1f3dde650ce7452fa9d41a5318b8df9f(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 20:54:01,493 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for f70bfe945d0054ce5a228cf255d902f5: 2023-04-19 20:54:01,493 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5., storeName=f70bfe945d0054ce5a228cf255d902f5/C, priority=12, startTime=1681937641393; duration=0sec 2023-04-19 20:54:01,493 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 20:54:01,505 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:54:01,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:54:01,508 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:54:01,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:54:01,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:54:01,510 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:54:01,510 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:01,510 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:54:01,510 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:54:01,511 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:01,511 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:01,513 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:01,513 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:01,513 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:01,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:01,514 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:01,514 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:54:01,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:01,514 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:01,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:54:01,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:01,515 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:54:01,515 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:54:01,515 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:54:01,515 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:54:01,515 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:01,515 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:54:01,516 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:54:01,516 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:54:01,516 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. 2023-04-19 20:54:01,516 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. started... 2023-04-19 20:54:01,516 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing f70bfe945d0054ce5a228cf255d902f5 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-19 20:54:01,516 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f70bfe945d0054ce5a228cf255d902f5, store=A 2023-04-19 20:54:01,516 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:54:01,516 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f70bfe945d0054ce5a228cf255d902f5, store=B 2023-04-19 20:54:01,516 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:54:01,516 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f70bfe945d0054ce5a228cf255d902f5, store=C 2023-04-19 20:54:01,517 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:54:01,526 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=883 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/A/c0325649019248849f473f7555dfb40a 2023-04-19 20:54:01,544 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=883 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/B/62a269254a4d404abb172dfdcdd81405 2023-04-19 20:54:01,559 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=883 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/C/099c8edaedbb45e4bbacef93dd6310a1 2023-04-19 20:54:01,568 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/A/c0325649019248849f473f7555dfb40a as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/c0325649019248849f473f7555dfb40a 2023-04-19 20:54:01,574 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/c0325649019248849f473f7555dfb40a, entries=150, sequenceid=883, filesize=11.9 K 2023-04-19 20:54:01,575 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/B/62a269254a4d404abb172dfdcdd81405 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/62a269254a4d404abb172dfdcdd81405 2023-04-19 20:54:01,581 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/62a269254a4d404abb172dfdcdd81405, entries=150, sequenceid=883, filesize=11.9 K 2023-04-19 20:54:01,584 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/C/099c8edaedbb45e4bbacef93dd6310a1 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/099c8edaedbb45e4bbacef93dd6310a1 2023-04-19 20:54:01,591 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/099c8edaedbb45e4bbacef93dd6310a1, entries=150, sequenceid=883, filesize=11.9 K 2023-04-19 20:54:01,599 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=0 B/0 for f70bfe945d0054ce5a228cf255d902f5 in 82ms, sequenceid=883, compaction requested=false 2023-04-19 20:54:01,599 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f70bfe945d0054ce5a228cf255d902f5: 2023-04-19 20:54:01,599 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. 2023-04-19 20:54:01,599 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:54:01,599 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:54:01,599 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:01,603 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:54:01,603 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:54:01,606 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:01,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:01,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:01,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:01,606 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:01,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:01,607 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:01,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:01,607 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:54:01,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:01,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:01,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:01,610 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:54:01,610 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:54:01,610 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@207cfdfe[Count = 0] remaining members to acquire global barrier 2023-04-19 20:54:01,610 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:01,612 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:01,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:01,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:01,612 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:54:01,612 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:54:01,612 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:54:01,612 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:01,612 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:54:01,614 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:54:01,614 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:01,614 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:54:01,614 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:01,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:01,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:01,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:01,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:01,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:01,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:01,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:01,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:01,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:01,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:01,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:54:01,621 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:54:01,621 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:54:01,621 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:54:01,621 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:54:01,621 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:54:01,621 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:54:01,624 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:01,624 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:01,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:01,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:01,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:01,624 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:01,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:01,624 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:01,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:01,625 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:01,625 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:01,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:01,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:01,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:01,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:01,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:01,626 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:01,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:01,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:01,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:01,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:01,629 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:01,629 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:01,629 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:01,629 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:01,629 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:01,629 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:54:01,629 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:01,629 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:54:01,629 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:01,629 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:01,629 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:01,629 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:01,630 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:01,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:01,630 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:54:01,630 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:54:01,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:01,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:01,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(9158): Flush requested on f70bfe945d0054ce5a228cf255d902f5 2023-04-19 20:54:01,633 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing f70bfe945d0054ce5a228cf255d902f5 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-19 20:54:01,634 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f70bfe945d0054ce5a228cf255d902f5, store=A 2023-04-19 20:54:01,635 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:54:01,635 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f70bfe945d0054ce5a228cf255d902f5, store=B 2023-04-19 20:54:01,635 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:54:01,635 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f70bfe945d0054ce5a228cf255d902f5, store=C 2023-04-19 20:54:01,635 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:54:01,662 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=895 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/A/c41403c5e523408daa7be5b5da11a219 2023-04-19 20:54:01,667 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:01,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 298 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42342 deadline: 1681937701662, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:01,668 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:01,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 296 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42380 deadline: 1681937701666, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:01,669 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:01,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 373 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42350 deadline: 1681937701667, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:01,674 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:01,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42384 deadline: 1681937701671, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:01,677 DEBUG [Thread-141] client.RpcRetryingCallerImpl(129): Call exception, tries=8, retries=16, started=18300 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5., hostname=jenkins-hbase4.apache.org,35247,1681937616971, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-19 20:54:01,730 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:54:01,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:54:01,733 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:54:01,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:54:01,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:54:01,734 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:54:01,734 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:01,734 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:54:01,734 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:54:01,734 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:01,734 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:01,744 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:01,744 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:01,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:01,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:01,744 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:01,744 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:54:01,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:01,745 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:01,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:54:01,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:01,745 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:54:01,745 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:54:01,745 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:54:01,746 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:54:01,746 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:01,746 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:54:01,746 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:54:01,746 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:54:01,746 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. 2023-04-19 20:54:01,746 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. started... 2023-04-19 20:54:01,746 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. as already flushing 2023-04-19 20:54:01,746 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. 2023-04-19 20:54:01,746 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:54:01,746 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:54:01,746 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:01,746 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:54:01,746 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:54:01,748 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:01,748 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:01,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:01,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:01,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:01,749 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:01,749 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:54:01,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:01,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:01,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:01,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:01,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:01,750 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:54:01,751 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2733255e[Count = 0] remaining members to acquire global barrier 2023-04-19 20:54:01,751 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:54:01,751 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:01,752 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:01,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:01,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:01,752 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:54:01,752 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:54:01,752 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:54:01,753 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:01,753 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:54:01,755 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:01,755 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:54:01,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:01,755 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:01,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:01,755 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:54:01,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:01,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:01,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:01,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:01,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:01,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:01,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:01,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:01,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:54:01,757 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:54:01,757 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:54:01,757 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:54:01,758 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:54:01,758 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:54:01,758 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:54:01,759 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:01,759 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:01,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:01,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:01,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:01,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:01,759 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:01,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:01,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:01,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:01,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:01,760 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:01,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:01,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:01,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:01,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:01,761 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:01,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:01,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:01,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:01,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:01,764 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:01,764 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:01,764 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:01,764 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:54:01,764 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:54:01,764 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:01,764 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:54:01,764 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:01,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:01,764 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:01,764 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:01,764 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:54:01,764 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:54:01,764 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:01,764 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:01,764 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:01,764 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:01,765 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:01,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:01,769 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:01,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 300 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42342 deadline: 1681937701768, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:01,771 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:01,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 298 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42380 deadline: 1681937701769, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:01,772 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:01,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 375 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42350 deadline: 1681937701770, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:01,831 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/B/e8023d5316b94e3aa00d132bd67e4e80 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/e8023d5316b94e3aa00d132bd67e4e80 2023-04-19 20:54:01,839 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in f70bfe945d0054ce5a228cf255d902f5/B of f70bfe945d0054ce5a228cf255d902f5 into e8023d5316b94e3aa00d132bd67e4e80(size=12.9 K), total size for store is 24.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 20:54:01,839 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for f70bfe945d0054ce5a228cf255d902f5: 2023-04-19 20:54:01,839 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5., storeName=f70bfe945d0054ce5a228cf255d902f5/B, priority=12, startTime=1681937641392; duration=0sec 2023-04-19 20:54:01,839 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 20:54:01,864 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:54:01,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:54:01,869 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:54:01,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:54:01,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:54:01,870 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:54:01,870 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:01,870 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:54:01,870 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:54:01,871 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:01,871 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:01,872 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:01,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:01,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:01,872 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:01,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:01,873 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:01,873 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:54:01,873 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:01,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:54:01,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:01,874 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:54:01,874 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:54:01,874 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:54:01,874 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:54:01,874 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:01,874 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:54:01,874 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:54:01,874 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:54:01,875 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. 2023-04-19 20:54:01,875 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. started... 2023-04-19 20:54:01,875 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. as already flushing 2023-04-19 20:54:01,875 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. 2023-04-19 20:54:01,875 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:54:01,875 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:54:01,875 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:01,875 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:54:01,875 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:54:01,880 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:01,880 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:01,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:01,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:01,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:01,881 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:01,881 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:54:01,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:01,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:01,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:01,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:01,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:01,885 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:54:01,885 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@519009a9[Count = 0] remaining members to acquire global barrier 2023-04-19 20:54:01,885 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:54:01,885 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:01,886 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:01,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:01,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:01,886 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:54:01,886 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:54:01,886 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:54:01,887 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:01,887 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:54:01,888 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:54:01,888 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:01,888 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:01,888 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:54:01,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:01,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:01,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:01,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:01,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:01,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:01,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:01,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:01,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:01,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:01,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:54:01,893 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:54:01,893 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:54:01,893 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:54:01,893 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:54:01,894 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:54:01,894 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:54:01,894 DEBUG [Thread-154] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5ae6c42b to 127.0.0.1:58736 2023-04-19 20:54:01,894 DEBUG [Thread-154] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-19 20:54:01,895 DEBUG [Thread-158] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6089dec1 to 127.0.0.1:58736 2023-04-19 20:54:01,895 DEBUG [Thread-158] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-19 20:54:01,895 DEBUG [Thread-152] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x29fba56a to 127.0.0.1:58736 2023-04-19 20:54:01,896 DEBUG [Thread-152] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-19 20:54:01,901 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:01,901 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:01,901 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:01,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:01,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:01,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:01,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:01,902 DEBUG [Thread-156] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1e44f1b3 to 127.0.0.1:58736 2023-04-19 20:54:01,902 DEBUG [Thread-156] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-19 20:54:01,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:01,972 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:01,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 302 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42342 deadline: 1681937701971, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:01,973 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:01,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 300 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42380 deadline: 1681937701973, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:01,973 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:01,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 377 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42350 deadline: 1681937701973, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:02,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:02,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:02,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:02,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:02,005 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:02,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:02,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:02,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:02,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:02,007 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:02,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:02,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:02,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:02,009 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:02,009 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:02,009 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:02,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:02,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:02,009 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:02,009 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:54:02,010 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:54:02,009 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:02,010 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:02,010 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:02,010 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:02,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:02,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:02,010 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:54:02,010 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:02,010 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:54:02,010 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:02,081 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=895 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/B/7d7cf57180c641f4aa49d346e799245d 2023-04-19 20:54:02,110 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:54:02,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:54:02,274 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:02,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 304 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42342 deadline: 1681937702274, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:02,274 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:02,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 302 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42380 deadline: 1681937702274, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:02,275 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:02,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 379 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42350 deadline: 1681937702275, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:02,492 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=895 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/C/fc1a16f2746542f6a37c77b54aba0f97 2023-04-19 20:54:02,497 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/A/c41403c5e523408daa7be5b5da11a219 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/c41403c5e523408daa7be5b5da11a219 2023-04-19 20:54:02,501 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/c41403c5e523408daa7be5b5da11a219, entries=150, sequenceid=895, filesize=11.9 K 2023-04-19 20:54:02,502 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/B/7d7cf57180c641f4aa49d346e799245d as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/7d7cf57180c641f4aa49d346e799245d 2023-04-19 20:54:02,506 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/7d7cf57180c641f4aa49d346e799245d, entries=150, sequenceid=895, filesize=11.9 K 2023-04-19 20:54:02,507 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/C/fc1a16f2746542f6a37c77b54aba0f97 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/fc1a16f2746542f6a37c77b54aba0f97 2023-04-19 20:54:02,511 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/fc1a16f2746542f6a37c77b54aba0f97, entries=150, sequenceid=895, filesize=11.9 K 2023-04-19 20:54:02,512 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for f70bfe945d0054ce5a228cf255d902f5 in 879ms, sequenceid=895, compaction requested=true 2023-04-19 20:54:02,512 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for f70bfe945d0054ce5a228cf255d902f5: 2023-04-19 20:54:02,512 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 20:54:02,512 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 20:54:02,512 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-19 20:54:02,512 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-19 20:54:02,512 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 20:54:02,514 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37546 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-19 20:54:02,514 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1912): f70bfe945d0054ce5a228cf255d902f5/A is initiating minor compaction (all files) 2023-04-19 20:54:02,514 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37546 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-19 20:54:02,514 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of f70bfe945d0054ce5a228cf255d902f5/A in TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. 2023-04-19 20:54:02,514 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1912): f70bfe945d0054ce5a228cf255d902f5/B is initiating minor compaction (all files) 2023-04-19 20:54:02,514 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/791cf76d266e4f2d9ccd8b4ad5f0d865, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/c0325649019248849f473f7555dfb40a, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/c41403c5e523408daa7be5b5da11a219] into tmpdir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp, totalSize=36.7 K 2023-04-19 20:54:02,514 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2259): Starting compaction of f70bfe945d0054ce5a228cf255d902f5/B in TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. 2023-04-19 20:54:02,514 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/e8023d5316b94e3aa00d132bd67e4e80, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/62a269254a4d404abb172dfdcdd81405, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/7d7cf57180c641f4aa49d346e799245d] into tmpdir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp, totalSize=36.7 K 2023-04-19 20:54:02,514 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 791cf76d266e4f2d9ccd8b4ad5f0d865, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=866, earliestPutTs=1681937639874 2023-04-19 20:54:02,514 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting e8023d5316b94e3aa00d132bd67e4e80, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=866, earliestPutTs=1681937639874 2023-04-19 20:54:02,515 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting c0325649019248849f473f7555dfb40a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=883, earliestPutTs=1681937640497 2023-04-19 20:54:02,515 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 62a269254a4d404abb172dfdcdd81405, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=883, earliestPutTs=1681937640497 2023-04-19 20:54:02,515 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting c41403c5e523408daa7be5b5da11a219, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=895, earliestPutTs=1681937641631 2023-04-19 20:54:02,515 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 7d7cf57180c641f4aa49d346e799245d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=895, earliestPutTs=1681937641631 2023-04-19 20:54:02,523 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] throttle.PressureAwareThroughputController(145): f70bfe945d0054ce5a228cf255d902f5#A#compaction#96 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 20:54:02,525 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] throttle.PressureAwareThroughputController(145): f70bfe945d0054ce5a228cf255d902f5#B#compaction#97 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 20:54:02,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(9158): Flush requested on f70bfe945d0054ce5a228cf255d902f5 2023-04-19 20:54:02,778 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing f70bfe945d0054ce5a228cf255d902f5 3/3 column families, dataSize=120.76 KB heapSize=360.05 KB 2023-04-19 20:54:02,778 DEBUG [Thread-145] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x401974c7 to 127.0.0.1:58736 2023-04-19 20:54:02,778 DEBUG [Thread-145] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-19 20:54:02,779 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681937641658, max timestamp=1681937642776Num uniques 859; 2023-04-19 20:54:02,779 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 20:54:02,779 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681937641658, max timestamp=1681937642776Num uniques 859; 2023-04-19 20:54:02,779 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 20:54:02,779 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681937641658, max timestamp=1681937642776Num uniques 859; 2023-04-19 20:54:02,779 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-19 20:54:02,779 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f70bfe945d0054ce5a228cf255d902f5, store=A 2023-04-19 20:54:02,779 DEBUG [Thread-147] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1ca4a062 to 127.0.0.1:58736 2023-04-19 20:54:02,780 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 20:54:02,780 DEBUG [Thread-147] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-19 20:54:02,780 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f70bfe945d0054ce5a228cf255d902f5, store=B 2023-04-19 20:54:02,780 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 20:54:02,780 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f70bfe945d0054ce5a228cf255d902f5, store=C 2023-04-19 20:54:02,780 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 20:54:02,781 DEBUG [Thread-143] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x377933e4 to 127.0.0.1:58736 2023-04-19 20:54:02,781 DEBUG [Thread-143] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-19 20:54:02,782 DEBUG [Thread-149] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6ec07124 to 127.0.0.1:58736 2023-04-19 20:54:02,782 DEBUG [Thread-149] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-19 20:54:02,940 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/B/b439f99174424efba8b5227b9190b4af as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/b439f99174424efba8b5227b9190b4af 2023-04-19 20:54:02,940 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/A/5d1ebbdf0d7f48eca8ca857acd85ceca as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/5d1ebbdf0d7f48eca8ca857acd85ceca 2023-04-19 20:54:02,946 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in f70bfe945d0054ce5a228cf255d902f5/A of f70bfe945d0054ce5a228cf255d902f5 into 5d1ebbdf0d7f48eca8ca857acd85ceca(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 20:54:02,946 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for f70bfe945d0054ce5a228cf255d902f5: 2023-04-19 20:54:02,946 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5., storeName=f70bfe945d0054ce5a228cf255d902f5/A, priority=13, startTime=1681937642512; duration=0sec 2023-04-19 20:54:02,946 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 20:54:02,946 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-19 20:54:02,947 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37546 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-19 20:54:02,947 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1912): f70bfe945d0054ce5a228cf255d902f5/C is initiating minor compaction (all files) 2023-04-19 20:54:02,948 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of f70bfe945d0054ce5a228cf255d902f5/C in TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. 2023-04-19 20:54:02,948 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/1f3dde650ce7452fa9d41a5318b8df9f, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/099c8edaedbb45e4bbacef93dd6310a1, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/fc1a16f2746542f6a37c77b54aba0f97] into tmpdir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp, totalSize=36.7 K 2023-04-19 20:54:02,948 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in f70bfe945d0054ce5a228cf255d902f5/B of f70bfe945d0054ce5a228cf255d902f5 into b439f99174424efba8b5227b9190b4af(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 20:54:02,948 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for f70bfe945d0054ce5a228cf255d902f5: 2023-04-19 20:54:02,948 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5., storeName=f70bfe945d0054ce5a228cf255d902f5/B, priority=13, startTime=1681937642512; duration=0sec 2023-04-19 20:54:02,948 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 1f3dde650ce7452fa9d41a5318b8df9f, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=866, earliestPutTs=1681937639874 2023-04-19 20:54:02,948 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 20:54:02,949 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 099c8edaedbb45e4bbacef93dd6310a1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=883, earliestPutTs=1681937640497 2023-04-19 20:54:02,949 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting fc1a16f2746542f6a37c77b54aba0f97, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=895, earliestPutTs=1681937641631 2023-04-19 20:54:02,956 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] throttle.PressureAwareThroughputController(145): f70bfe945d0054ce5a228cf255d902f5#C#compaction#99 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 20:54:02,972 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/C/c6571fbaea8e4afe8971633780fcbb4b as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/c6571fbaea8e4afe8971633780fcbb4b 2023-04-19 20:54:02,979 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in f70bfe945d0054ce5a228cf255d902f5/C of f70bfe945d0054ce5a228cf255d902f5 into c6571fbaea8e4afe8971633780fcbb4b(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 20:54:02,979 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for f70bfe945d0054ce5a228cf255d902f5: 2023-04-19 20:54:02,979 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5., storeName=f70bfe945d0054ce5a228cf255d902f5/C, priority=13, startTime=1681937642512; duration=0sec 2023-04-19 20:54:02,979 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 20:54:03,203 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=918 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/A/a1428e34e0eb4851ad9fab11ae478944 2023-04-19 20:54:03,614 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=918 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/B/826baf3e3a7740b99a7c84d850c5deba 2023-04-19 20:54:04,025 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=10.79 KB at sequenceid=918 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/C/f232dc3763704f0cba7e17c604e3990a 2023-04-19 20:54:04,030 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/A/a1428e34e0eb4851ad9fab11ae478944 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/a1428e34e0eb4851ad9fab11ae478944 2023-04-19 20:54:04,034 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/a1428e34e0eb4851ad9fab11ae478944, entries=150, sequenceid=918, filesize=11.9 K 2023-04-19 20:54:04,035 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/B/826baf3e3a7740b99a7c84d850c5deba as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/826baf3e3a7740b99a7c84d850c5deba 2023-04-19 20:54:04,039 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/826baf3e3a7740b99a7c84d850c5deba, entries=150, sequenceid=918, filesize=11.9 K 2023-04-19 20:54:04,040 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/C/f232dc3763704f0cba7e17c604e3990a as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/f232dc3763704f0cba7e17c604e3990a 2023-04-19 20:54:04,044 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/f232dc3763704f0cba7e17c604e3990a, entries=150, sequenceid=918, filesize=11.9 K 2023-04-19 20:54:04,045 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~33.15 KB/33945, heapSize ~85.20 KB/87240, currentSize=13.42 KB/13740 for f70bfe945d0054ce5a228cf255d902f5 in 1267ms, sequenceid=918, compaction requested=false 2023-04-19 20:54:04,045 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for f70bfe945d0054ce5a228cf255d902f5: 2023-04-19 20:54:11,684 DEBUG [Thread-141] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x402c2635 to 127.0.0.1:58736 2023-04-19 20:54:11,684 DEBUG [Thread-141] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-19 20:54:11,685 INFO [Listener at localhost/34893] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-19 20:54:11,685 INFO [Listener at localhost/34893] hbase.AcidGuaranteesTestTool(394): wrote 86 2023-04-19 20:54:11,685 INFO [Listener at localhost/34893] hbase.AcidGuaranteesTestTool(394): wrote 231 2023-04-19 20:54:11,685 INFO [Listener at localhost/34893] hbase.AcidGuaranteesTestTool(394): wrote 178 2023-04-19 20:54:11,685 INFO [Listener at localhost/34893] hbase.AcidGuaranteesTestTool(394): wrote 180 2023-04-19 20:54:11,685 INFO [Listener at localhost/34893] hbase.AcidGuaranteesTestTool(394): wrote 146 2023-04-19 20:54:11,685 INFO [Listener at localhost/34893] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-19 20:54:11,685 INFO [Listener at localhost/34893] hbase.AcidGuaranteesTestTool(398): read 3742 2023-04-19 20:54:11,685 INFO [Listener at localhost/34893] hbase.AcidGuaranteesTestTool(398): read 3736 2023-04-19 20:54:11,685 INFO [Listener at localhost/34893] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-19 20:54:11,685 INFO [Listener at localhost/34893] hbase.AcidGuaranteesTestTool(402): scanned 1669 2023-04-19 20:54:11,686 INFO [Listener at localhost/34893] hbase.AcidGuaranteesTestTool(403): verified 4991 rows 2023-04-19 20:54:11,686 INFO [Listener at localhost/34893] hbase.AcidGuaranteesTestTool(402): scanned 1669 2023-04-19 20:54:11,686 INFO [Listener at localhost/34893] hbase.AcidGuaranteesTestTool(403): verified 5000 rows 2023-04-19 20:54:11,686 INFO [Listener at localhost/34893] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-19 20:54:11,686 DEBUG [Listener at localhost/34893] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2e0a05b0 to 127.0.0.1:58736 2023-04-19 20:54:11,686 DEBUG [Listener at localhost/34893] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-19 20:54:11,691 INFO [Listener at localhost/34893] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-19 20:54:11,698 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-19 20:54:11,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure2.ProcedureExecutor(1029): Stored pid=12, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-19 20:54:11,711 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681937651710"}]},"ts":"1681937651710"} 2023-04-19 20:54:11,712 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-19 20:54:11,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-19 20:54:11,714 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-19 20:54:11,715 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=f70bfe945d0054ce5a228cf255d902f5, UNASSIGN}] 2023-04-19 20:54:11,717 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=f70bfe945d0054ce5a228cf255d902f5, UNASSIGN 2023-04-19 20:54:11,718 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=f70bfe945d0054ce5a228cf255d902f5, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:11,718 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681937651718"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681937651718"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681937651718"}]},"ts":"1681937651718"} 2023-04-19 20:54:11,720 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=14, ppid=13, state=RUNNABLE; CloseRegionProcedure f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971}] 2023-04-19 20:54:11,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-19 20:54:11,878 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close f70bfe945d0054ce5a228cf255d902f5 2023-04-19 20:54:11,878 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing f70bfe945d0054ce5a228cf255d902f5, disabling compactions & flushes 2023-04-19 20:54:11,878 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. 2023-04-19 20:54:11,878 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. 2023-04-19 20:54:11,878 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. after waiting 0 ms 2023-04-19 20:54:11,878 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. 2023-04-19 20:54:11,878 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing f70bfe945d0054ce5a228cf255d902f5 3/3 column families, dataSize=20.13 KB heapSize=60.52 KB 2023-04-19 20:54:11,879 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f70bfe945d0054ce5a228cf255d902f5, store=A 2023-04-19 20:54:11,879 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:54:11,879 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f70bfe945d0054ce5a228cf255d902f5, store=B 2023-04-19 20:54:11,879 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:54:11,879 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f70bfe945d0054ce5a228cf255d902f5, store=C 2023-04-19 20:54:11,879 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:54:12,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-19 20:54:12,287 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=927 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/A/c0a3e13306374b269a879c7ad89612d8 2023-04-19 20:54:12,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-19 20:54:12,709 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=927 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/B/9922e40eca184939bae2bd5bcc404d16 2023-04-19 20:54:12,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-19 20:54:13,120 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=927 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/C/a2c8f0d3a0444f69b26a95575311a0cb 2023-04-19 20:54:13,125 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/A/c0a3e13306374b269a879c7ad89612d8 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/c0a3e13306374b269a879c7ad89612d8 2023-04-19 20:54:13,131 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/c0a3e13306374b269a879c7ad89612d8, entries=150, sequenceid=927, filesize=11.9 K 2023-04-19 20:54:13,132 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/B/9922e40eca184939bae2bd5bcc404d16 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/9922e40eca184939bae2bd5bcc404d16 2023-04-19 20:54:13,138 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/9922e40eca184939bae2bd5bcc404d16, entries=150, sequenceid=927, filesize=11.9 K 2023-04-19 20:54:13,139 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/.tmp/C/a2c8f0d3a0444f69b26a95575311a0cb as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/a2c8f0d3a0444f69b26a95575311a0cb 2023-04-19 20:54:13,144 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/a2c8f0d3a0444f69b26a95575311a0cb, entries=150, sequenceid=927, filesize=11.9 K 2023-04-19 20:54:13,145 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.13 KB/20610, heapSize ~60.47 KB/61920, currentSize=0 B/0 for f70bfe945d0054ce5a228cf255d902f5 in 1267ms, sequenceid=927, compaction requested=true 2023-04-19 20:54:13,165 DEBUG [StoreCloser-TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/c5820d14b7d445a39e49141037c74f93, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/6d7e233041aa47a8a61054b74f3c78e3, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/30020030f05841caa3cd673c2421ca53, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/f08f567cfb31466880001902682269e7, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/310510f59e1944fd9fd6b7481cb2cbed, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/acfc472dda4a49979cd46ad6194df803, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/f17c859230b84ccb8b0da865f1285cf2, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/ff28f839fa734083a05196ca36963ce7, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/f7ce0cda94e94edc83f1dbcf54f4e122, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/361fbc254d34492ea61eff57284aec32, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/15fe198edcbf437195057b1b1ac62025, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/5627443d993f4f9487601963a30ea217, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/c7899702bdf34f54862407fd3c7dcc91, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/5860593eba5b479fb5f432352eca0a37, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/dc9ee4d0171144e690af5026300de9cd, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/9266d1eeb42b440eaa5084e68480614a, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/78780e4b985840c18ce9ab9115c522dd, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/4d6f3b70db424f4a84619f347aedb87b, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/b0fe4da425c746a193066d70af799d1a, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/9b3d5fd0cd044e248b6db6802c4ae0d4, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/044073d919ed4b08908d57b46c1885e0, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/7f20fbd572a54adc92250879c385c67e, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/9890432f65ab43ec80ca6142349e6f56, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/992f11e8de07463186c620f55d66fbf2, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/5b8f8921d07d4afab18158aee72a417d, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/ccf5eaa3d0f04dc598d3f0b104e1ffdf, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/95b8703e92284865bb446131803df1c4, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/3ee37a4c3c004b9cb346b73b5d6177b3, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/4fda88ae9c5b46c0a5f24c6c1cba1231, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/791cf76d266e4f2d9ccd8b4ad5f0d865, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/c0325649019248849f473f7555dfb40a, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/c41403c5e523408daa7be5b5da11a219] to archive 2023-04-19 20:54:13,166 DEBUG [StoreCloser-TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-19 20:54:13,172 DEBUG [StoreCloser-TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/c5820d14b7d445a39e49141037c74f93 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/c5820d14b7d445a39e49141037c74f93 2023-04-19 20:54:13,173 DEBUG [StoreCloser-TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/6d7e233041aa47a8a61054b74f3c78e3 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/6d7e233041aa47a8a61054b74f3c78e3 2023-04-19 20:54:13,175 DEBUG [StoreCloser-TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/30020030f05841caa3cd673c2421ca53 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/30020030f05841caa3cd673c2421ca53 2023-04-19 20:54:13,176 DEBUG [StoreCloser-TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/f08f567cfb31466880001902682269e7 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/f08f567cfb31466880001902682269e7 2023-04-19 20:54:13,178 DEBUG [StoreCloser-TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/310510f59e1944fd9fd6b7481cb2cbed to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/310510f59e1944fd9fd6b7481cb2cbed 2023-04-19 20:54:13,180 DEBUG [StoreCloser-TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/acfc472dda4a49979cd46ad6194df803 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/acfc472dda4a49979cd46ad6194df803 2023-04-19 20:54:13,181 DEBUG [StoreCloser-TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/f17c859230b84ccb8b0da865f1285cf2 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/f17c859230b84ccb8b0da865f1285cf2 2023-04-19 20:54:13,183 DEBUG [StoreCloser-TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/ff28f839fa734083a05196ca36963ce7 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/ff28f839fa734083a05196ca36963ce7 2023-04-19 20:54:13,184 DEBUG [StoreCloser-TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/f7ce0cda94e94edc83f1dbcf54f4e122 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/f7ce0cda94e94edc83f1dbcf54f4e122 2023-04-19 20:54:13,185 DEBUG [StoreCloser-TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/361fbc254d34492ea61eff57284aec32 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/361fbc254d34492ea61eff57284aec32 2023-04-19 20:54:13,186 DEBUG [StoreCloser-TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/15fe198edcbf437195057b1b1ac62025 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/15fe198edcbf437195057b1b1ac62025 2023-04-19 20:54:13,187 DEBUG [StoreCloser-TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/5627443d993f4f9487601963a30ea217 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/5627443d993f4f9487601963a30ea217 2023-04-19 20:54:13,189 DEBUG [StoreCloser-TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/c7899702bdf34f54862407fd3c7dcc91 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/c7899702bdf34f54862407fd3c7dcc91 2023-04-19 20:54:13,190 DEBUG [StoreCloser-TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/5860593eba5b479fb5f432352eca0a37 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/5860593eba5b479fb5f432352eca0a37 2023-04-19 20:54:13,191 DEBUG [StoreCloser-TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/dc9ee4d0171144e690af5026300de9cd to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/dc9ee4d0171144e690af5026300de9cd 2023-04-19 20:54:13,193 DEBUG [StoreCloser-TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/9266d1eeb42b440eaa5084e68480614a to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/9266d1eeb42b440eaa5084e68480614a 2023-04-19 20:54:13,194 DEBUG [StoreCloser-TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/78780e4b985840c18ce9ab9115c522dd to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/78780e4b985840c18ce9ab9115c522dd 2023-04-19 20:54:13,195 DEBUG [StoreCloser-TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/4d6f3b70db424f4a84619f347aedb87b to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/4d6f3b70db424f4a84619f347aedb87b 2023-04-19 20:54:13,197 DEBUG [StoreCloser-TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/b0fe4da425c746a193066d70af799d1a to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/b0fe4da425c746a193066d70af799d1a 2023-04-19 20:54:13,198 DEBUG [StoreCloser-TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/9b3d5fd0cd044e248b6db6802c4ae0d4 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/9b3d5fd0cd044e248b6db6802c4ae0d4 2023-04-19 20:54:13,199 DEBUG [StoreCloser-TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/044073d919ed4b08908d57b46c1885e0 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/044073d919ed4b08908d57b46c1885e0 2023-04-19 20:54:13,200 DEBUG [StoreCloser-TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/7f20fbd572a54adc92250879c385c67e to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/7f20fbd572a54adc92250879c385c67e 2023-04-19 20:54:13,201 DEBUG [StoreCloser-TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/9890432f65ab43ec80ca6142349e6f56 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/9890432f65ab43ec80ca6142349e6f56 2023-04-19 20:54:13,202 DEBUG [StoreCloser-TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/992f11e8de07463186c620f55d66fbf2 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/992f11e8de07463186c620f55d66fbf2 2023-04-19 20:54:13,203 DEBUG [StoreCloser-TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/5b8f8921d07d4afab18158aee72a417d to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/5b8f8921d07d4afab18158aee72a417d 2023-04-19 20:54:13,204 DEBUG [StoreCloser-TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/ccf5eaa3d0f04dc598d3f0b104e1ffdf to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/ccf5eaa3d0f04dc598d3f0b104e1ffdf 2023-04-19 20:54:13,205 DEBUG [StoreCloser-TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/95b8703e92284865bb446131803df1c4 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/95b8703e92284865bb446131803df1c4 2023-04-19 20:54:13,206 DEBUG [StoreCloser-TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/3ee37a4c3c004b9cb346b73b5d6177b3 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/3ee37a4c3c004b9cb346b73b5d6177b3 2023-04-19 20:54:13,207 DEBUG [StoreCloser-TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/4fda88ae9c5b46c0a5f24c6c1cba1231 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/4fda88ae9c5b46c0a5f24c6c1cba1231 2023-04-19 20:54:13,208 DEBUG [StoreCloser-TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/791cf76d266e4f2d9ccd8b4ad5f0d865 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/791cf76d266e4f2d9ccd8b4ad5f0d865 2023-04-19 20:54:13,209 DEBUG [StoreCloser-TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/c0325649019248849f473f7555dfb40a to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/c0325649019248849f473f7555dfb40a 2023-04-19 20:54:13,210 DEBUG [StoreCloser-TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/c41403c5e523408daa7be5b5da11a219 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/c41403c5e523408daa7be5b5da11a219 2023-04-19 20:54:13,233 DEBUG [StoreCloser-TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/daf9346af1d1405c96ef9a592f36872e, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/f02035798baf4d76a1a8ef590e665f2e, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/046686358c124f62a93872cd1f9af3fc, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/98d3753be8bf4318b1a507e913e98d3a, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/0cfcc2ef2a8e4ed681330504c0cdc822, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/cc0c912afae348bc9eabbc3730a4e297, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/e158ae9bbf07487b8bdb88909892fd88, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/e8ee299c3faa47798edd5710a18a826f, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/5c68103d0c30480bac080a8759341ce1, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/10a9f31a376a40e39a39033d457e01c4, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/1cb11e3b9ab540228aae33bb7ac4cc46, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/b0da787dd1ca42f89b336b3dbd2d12c5, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/6d00dd51967e4100b40f46aec06d43b7, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/8a009e695e884b3c849580f60c78f004, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/aef57242c1d54d128a4795354e0dc257, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/e7b5f69e860f4c1aa8d50776099e1538, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/7f29115fa18c49d18d69a6ea9ac024a7, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/b7a46c57b6be4fc1bbf2ec6393740cbb, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/c09cb8f541374249b000d1974a826a55, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/f47c98c5c223486a9d083bfdf20de744, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/5949adbab09645cfa3cf825e42a09599, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/6f60ac93c4f349848df99b8fa81dc88a, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/62282af0f5144651a6b3d54b7c733b36, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/1b535ff8dc764876bdea11aafd523c70, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/fa7ebf4413e5438d84e156608c64a1ec, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/15887afc4d5947bd9c14d93ec5e8a98e, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/bdc9fcc38d60479498d1a84e4de8f269, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/d9e6eca8675c4dabb234ea1489df690c, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/e8023d5316b94e3aa00d132bd67e4e80, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/8b66d3b9abca4e21872bee111873d51a, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/62a269254a4d404abb172dfdcdd81405, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/7d7cf57180c641f4aa49d346e799245d] to archive 2023-04-19 20:54:13,234 DEBUG [StoreCloser-TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-19 20:54:13,235 DEBUG [StoreCloser-TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/daf9346af1d1405c96ef9a592f36872e to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/daf9346af1d1405c96ef9a592f36872e 2023-04-19 20:54:13,236 DEBUG [StoreCloser-TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/f02035798baf4d76a1a8ef590e665f2e to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/f02035798baf4d76a1a8ef590e665f2e 2023-04-19 20:54:13,237 DEBUG [StoreCloser-TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/046686358c124f62a93872cd1f9af3fc to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/046686358c124f62a93872cd1f9af3fc 2023-04-19 20:54:13,238 DEBUG [StoreCloser-TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/98d3753be8bf4318b1a507e913e98d3a to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/98d3753be8bf4318b1a507e913e98d3a 2023-04-19 20:54:13,239 DEBUG [StoreCloser-TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/0cfcc2ef2a8e4ed681330504c0cdc822 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/0cfcc2ef2a8e4ed681330504c0cdc822 2023-04-19 20:54:13,240 DEBUG [StoreCloser-TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/cc0c912afae348bc9eabbc3730a4e297 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/cc0c912afae348bc9eabbc3730a4e297 2023-04-19 20:54:13,241 DEBUG [StoreCloser-TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/e158ae9bbf07487b8bdb88909892fd88 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/e158ae9bbf07487b8bdb88909892fd88 2023-04-19 20:54:13,242 DEBUG [StoreCloser-TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/e8ee299c3faa47798edd5710a18a826f to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/e8ee299c3faa47798edd5710a18a826f 2023-04-19 20:54:13,243 DEBUG [StoreCloser-TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/5c68103d0c30480bac080a8759341ce1 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/5c68103d0c30480bac080a8759341ce1 2023-04-19 20:54:13,244 DEBUG [StoreCloser-TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/10a9f31a376a40e39a39033d457e01c4 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/10a9f31a376a40e39a39033d457e01c4 2023-04-19 20:54:13,245 DEBUG [StoreCloser-TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/1cb11e3b9ab540228aae33bb7ac4cc46 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/1cb11e3b9ab540228aae33bb7ac4cc46 2023-04-19 20:54:13,246 DEBUG [StoreCloser-TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/b0da787dd1ca42f89b336b3dbd2d12c5 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/b0da787dd1ca42f89b336b3dbd2d12c5 2023-04-19 20:54:13,247 DEBUG [StoreCloser-TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/6d00dd51967e4100b40f46aec06d43b7 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/6d00dd51967e4100b40f46aec06d43b7 2023-04-19 20:54:13,248 DEBUG [StoreCloser-TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/8a009e695e884b3c849580f60c78f004 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/8a009e695e884b3c849580f60c78f004 2023-04-19 20:54:13,249 DEBUG [StoreCloser-TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/aef57242c1d54d128a4795354e0dc257 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/aef57242c1d54d128a4795354e0dc257 2023-04-19 20:54:13,250 DEBUG [StoreCloser-TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/e7b5f69e860f4c1aa8d50776099e1538 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/e7b5f69e860f4c1aa8d50776099e1538 2023-04-19 20:54:13,251 DEBUG [StoreCloser-TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/7f29115fa18c49d18d69a6ea9ac024a7 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/7f29115fa18c49d18d69a6ea9ac024a7 2023-04-19 20:54:13,253 DEBUG [StoreCloser-TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/b7a46c57b6be4fc1bbf2ec6393740cbb to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/b7a46c57b6be4fc1bbf2ec6393740cbb 2023-04-19 20:54:13,254 DEBUG [StoreCloser-TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/c09cb8f541374249b000d1974a826a55 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/c09cb8f541374249b000d1974a826a55 2023-04-19 20:54:13,255 DEBUG [StoreCloser-TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/f47c98c5c223486a9d083bfdf20de744 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/f47c98c5c223486a9d083bfdf20de744 2023-04-19 20:54:13,256 DEBUG [StoreCloser-TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/5949adbab09645cfa3cf825e42a09599 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/5949adbab09645cfa3cf825e42a09599 2023-04-19 20:54:13,257 DEBUG [StoreCloser-TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/6f60ac93c4f349848df99b8fa81dc88a to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/6f60ac93c4f349848df99b8fa81dc88a 2023-04-19 20:54:13,258 DEBUG [StoreCloser-TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/62282af0f5144651a6b3d54b7c733b36 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/62282af0f5144651a6b3d54b7c733b36 2023-04-19 20:54:13,259 DEBUG [StoreCloser-TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/1b535ff8dc764876bdea11aafd523c70 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/1b535ff8dc764876bdea11aafd523c70 2023-04-19 20:54:13,260 DEBUG [StoreCloser-TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/fa7ebf4413e5438d84e156608c64a1ec to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/fa7ebf4413e5438d84e156608c64a1ec 2023-04-19 20:54:13,261 DEBUG [StoreCloser-TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/15887afc4d5947bd9c14d93ec5e8a98e to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/15887afc4d5947bd9c14d93ec5e8a98e 2023-04-19 20:54:13,262 DEBUG [StoreCloser-TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/bdc9fcc38d60479498d1a84e4de8f269 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/bdc9fcc38d60479498d1a84e4de8f269 2023-04-19 20:54:13,264 DEBUG [StoreCloser-TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/d9e6eca8675c4dabb234ea1489df690c to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/d9e6eca8675c4dabb234ea1489df690c 2023-04-19 20:54:13,265 DEBUG [StoreCloser-TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/e8023d5316b94e3aa00d132bd67e4e80 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/e8023d5316b94e3aa00d132bd67e4e80 2023-04-19 20:54:13,266 DEBUG [StoreCloser-TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/8b66d3b9abca4e21872bee111873d51a to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/8b66d3b9abca4e21872bee111873d51a 2023-04-19 20:54:13,268 DEBUG [StoreCloser-TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/62a269254a4d404abb172dfdcdd81405 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/62a269254a4d404abb172dfdcdd81405 2023-04-19 20:54:13,269 DEBUG [StoreCloser-TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/7d7cf57180c641f4aa49d346e799245d to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/7d7cf57180c641f4aa49d346e799245d 2023-04-19 20:54:13,273 DEBUG [StoreCloser-TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/0fb95bdc773047e3b2042cbe5f94f3c0, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/14d26d1a0f2749e18f2758e1568b64fb, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/5c18a7d2e1864b22ba9545a0f8bb6b8a, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/be7f3614a1194d0cbf874797b1d736cd, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/78ae43b6947e4c41a8a8fe9c8f556631, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/492a54bd908348119b1b63b030bee115, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/d397c066eb50460e97d2137e50678dc5, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/5e1159c20c7a468d8890a6be2b8535b1, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/b6e99014e8c34df7b74195b8cba735f0, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/3167eafbcb14456aa5deb5ff4d6e0fa0, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/575d80cfd03d415cadd11d98cea046d5, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/bea3bab17cb740d08624752ba4a9203c, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/69f71eb7bce9460fa12e370dce2a1ccb, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/6f0002a2b36d44f2aff335d297fbdae3, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/3f3b84850e0344d0aff253a68946da07, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/fc84d90f0d544b1dab4c05b7edfd112f, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/9a206002c16642d3bd46da50c12a4c8d, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/34214c7d26fa4e4082a99c85a1f29782, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/9407d1254a5c43de9a87022d7daf7a22, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/9c7eb29c72d243b2821225121fba85ba, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/fabfaf48242b46048e1d5ab2cd87c458, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/182d5ab2d07a4cae817fe22555780f69, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/6faed4d004564a5da0f1773f2b8fc1b6, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/3b3520a3aa9a4571b8953d385143b0a6, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/4a626fd6bf424606b77ae02cf5c7697f, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/368c297852e643b79702620e0faf327b, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/706dc5a4d58e4ef790b9a621091cf852, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/ea802b8e8b0e4d2e836df64a749947ee, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/1f3dde650ce7452fa9d41a5318b8df9f, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/09b23b0533b94d2b99a56701c9476800, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/099c8edaedbb45e4bbacef93dd6310a1, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/fc1a16f2746542f6a37c77b54aba0f97] to archive 2023-04-19 20:54:13,273 DEBUG [StoreCloser-TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-19 20:54:13,275 DEBUG [StoreCloser-TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/0fb95bdc773047e3b2042cbe5f94f3c0 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/0fb95bdc773047e3b2042cbe5f94f3c0 2023-04-19 20:54:13,276 DEBUG [StoreCloser-TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/14d26d1a0f2749e18f2758e1568b64fb to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/14d26d1a0f2749e18f2758e1568b64fb 2023-04-19 20:54:13,278 DEBUG [StoreCloser-TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/5c18a7d2e1864b22ba9545a0f8bb6b8a to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/5c18a7d2e1864b22ba9545a0f8bb6b8a 2023-04-19 20:54:13,279 DEBUG [StoreCloser-TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/be7f3614a1194d0cbf874797b1d736cd to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/be7f3614a1194d0cbf874797b1d736cd 2023-04-19 20:54:13,280 DEBUG [StoreCloser-TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/78ae43b6947e4c41a8a8fe9c8f556631 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/78ae43b6947e4c41a8a8fe9c8f556631 2023-04-19 20:54:13,282 DEBUG [StoreCloser-TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/492a54bd908348119b1b63b030bee115 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/492a54bd908348119b1b63b030bee115 2023-04-19 20:54:13,283 DEBUG [StoreCloser-TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/d397c066eb50460e97d2137e50678dc5 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/d397c066eb50460e97d2137e50678dc5 2023-04-19 20:54:13,284 DEBUG [StoreCloser-TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/5e1159c20c7a468d8890a6be2b8535b1 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/5e1159c20c7a468d8890a6be2b8535b1 2023-04-19 20:54:13,286 DEBUG [StoreCloser-TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/b6e99014e8c34df7b74195b8cba735f0 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/b6e99014e8c34df7b74195b8cba735f0 2023-04-19 20:54:13,287 DEBUG [StoreCloser-TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/3167eafbcb14456aa5deb5ff4d6e0fa0 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/3167eafbcb14456aa5deb5ff4d6e0fa0 2023-04-19 20:54:13,289 DEBUG [StoreCloser-TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/575d80cfd03d415cadd11d98cea046d5 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/575d80cfd03d415cadd11d98cea046d5 2023-04-19 20:54:13,291 DEBUG [StoreCloser-TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/bea3bab17cb740d08624752ba4a9203c to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/bea3bab17cb740d08624752ba4a9203c 2023-04-19 20:54:13,293 DEBUG [StoreCloser-TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/69f71eb7bce9460fa12e370dce2a1ccb to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/69f71eb7bce9460fa12e370dce2a1ccb 2023-04-19 20:54:13,294 DEBUG [StoreCloser-TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/6f0002a2b36d44f2aff335d297fbdae3 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/6f0002a2b36d44f2aff335d297fbdae3 2023-04-19 20:54:13,296 DEBUG [StoreCloser-TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/3f3b84850e0344d0aff253a68946da07 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/3f3b84850e0344d0aff253a68946da07 2023-04-19 20:54:13,297 DEBUG [StoreCloser-TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/fc84d90f0d544b1dab4c05b7edfd112f to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/fc84d90f0d544b1dab4c05b7edfd112f 2023-04-19 20:54:13,299 DEBUG [StoreCloser-TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/9a206002c16642d3bd46da50c12a4c8d to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/9a206002c16642d3bd46da50c12a4c8d 2023-04-19 20:54:13,300 DEBUG [StoreCloser-TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/34214c7d26fa4e4082a99c85a1f29782 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/34214c7d26fa4e4082a99c85a1f29782 2023-04-19 20:54:13,302 DEBUG [StoreCloser-TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/9407d1254a5c43de9a87022d7daf7a22 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/9407d1254a5c43de9a87022d7daf7a22 2023-04-19 20:54:13,303 DEBUG [StoreCloser-TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/9c7eb29c72d243b2821225121fba85ba to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/9c7eb29c72d243b2821225121fba85ba 2023-04-19 20:54:13,304 DEBUG [StoreCloser-TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/fabfaf48242b46048e1d5ab2cd87c458 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/fabfaf48242b46048e1d5ab2cd87c458 2023-04-19 20:54:13,306 DEBUG [StoreCloser-TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/182d5ab2d07a4cae817fe22555780f69 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/182d5ab2d07a4cae817fe22555780f69 2023-04-19 20:54:13,308 DEBUG [StoreCloser-TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/6faed4d004564a5da0f1773f2b8fc1b6 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/6faed4d004564a5da0f1773f2b8fc1b6 2023-04-19 20:54:13,309 DEBUG [StoreCloser-TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/3b3520a3aa9a4571b8953d385143b0a6 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/3b3520a3aa9a4571b8953d385143b0a6 2023-04-19 20:54:13,310 DEBUG [StoreCloser-TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/4a626fd6bf424606b77ae02cf5c7697f to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/4a626fd6bf424606b77ae02cf5c7697f 2023-04-19 20:54:13,311 DEBUG [StoreCloser-TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/368c297852e643b79702620e0faf327b to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/368c297852e643b79702620e0faf327b 2023-04-19 20:54:13,312 DEBUG [StoreCloser-TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/706dc5a4d58e4ef790b9a621091cf852 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/706dc5a4d58e4ef790b9a621091cf852 2023-04-19 20:54:13,314 DEBUG [StoreCloser-TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/ea802b8e8b0e4d2e836df64a749947ee to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/ea802b8e8b0e4d2e836df64a749947ee 2023-04-19 20:54:13,315 DEBUG [StoreCloser-TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/1f3dde650ce7452fa9d41a5318b8df9f to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/1f3dde650ce7452fa9d41a5318b8df9f 2023-04-19 20:54:13,315 DEBUG [StoreCloser-TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/09b23b0533b94d2b99a56701c9476800 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/09b23b0533b94d2b99a56701c9476800 2023-04-19 20:54:13,316 DEBUG [StoreCloser-TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/099c8edaedbb45e4bbacef93dd6310a1 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/099c8edaedbb45e4bbacef93dd6310a1 2023-04-19 20:54:13,317 DEBUG [StoreCloser-TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/fc1a16f2746542f6a37c77b54aba0f97 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/fc1a16f2746542f6a37c77b54aba0f97 2023-04-19 20:54:13,322 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/recovered.edits/930.seqid, newMaxSeqId=930, maxSeqId=1 2023-04-19 20:54:13,324 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5. 2023-04-19 20:54:13,324 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for f70bfe945d0054ce5a228cf255d902f5: 2023-04-19 20:54:13,325 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed f70bfe945d0054ce5a228cf255d902f5 2023-04-19 20:54:13,326 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=f70bfe945d0054ce5a228cf255d902f5, regionState=CLOSED 2023-04-19 20:54:13,326 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681937653326"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681937653326"}]},"ts":"1681937653326"} 2023-04-19 20:54:13,329 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=14, resume processing ppid=13 2023-04-19 20:54:13,329 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=14, ppid=13, state=SUCCESS; CloseRegionProcedure f70bfe945d0054ce5a228cf255d902f5, server=jenkins-hbase4.apache.org,35247,1681937616971 in 1.6070 sec 2023-04-19 20:54:13,330 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=13, resume processing ppid=12 2023-04-19 20:54:13,330 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=13, ppid=12, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=f70bfe945d0054ce5a228cf255d902f5, UNASSIGN in 1.6140 sec 2023-04-19 20:54:13,331 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681937653330"}]},"ts":"1681937653330"} 2023-04-19 20:54:13,331 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-19 20:54:13,333 INFO [PEWorker-4] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-19 20:54:13,335 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=12, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.6340 sec 2023-04-19 20:54:13,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-19 20:54:13,817 INFO [Listener at localhost/34893] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 12 completed 2023-04-19 20:54:13,821 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-19 20:54:13,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure2.ProcedureExecutor(1029): Stored pid=15, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-19 20:54:13,830 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=15, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-19 20:54:13,832 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=15, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-19 20:54:13,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-04-19 20:54:13,836 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/.tmp/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5 2023-04-19 20:54:13,840 DEBUG [HFileArchiver-3] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/.tmp/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A, FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/.tmp/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B, FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/.tmp/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C, FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/.tmp/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/recovered.edits] 2023-04-19 20:54:13,843 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/.tmp/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/5d1ebbdf0d7f48eca8ca857acd85ceca to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/5d1ebbdf0d7f48eca8ca857acd85ceca 2023-04-19 20:54:13,844 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/.tmp/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/a1428e34e0eb4851ad9fab11ae478944 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/a1428e34e0eb4851ad9fab11ae478944 2023-04-19 20:54:13,845 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/.tmp/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/c0a3e13306374b269a879c7ad89612d8 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/A/c0a3e13306374b269a879c7ad89612d8 2023-04-19 20:54:13,848 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/.tmp/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/826baf3e3a7740b99a7c84d850c5deba to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/826baf3e3a7740b99a7c84d850c5deba 2023-04-19 20:54:13,849 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/.tmp/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/9922e40eca184939bae2bd5bcc404d16 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/9922e40eca184939bae2bd5bcc404d16 2023-04-19 20:54:13,850 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/.tmp/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/b439f99174424efba8b5227b9190b4af to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/B/b439f99174424efba8b5227b9190b4af 2023-04-19 20:54:13,853 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/.tmp/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/a2c8f0d3a0444f69b26a95575311a0cb to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/a2c8f0d3a0444f69b26a95575311a0cb 2023-04-19 20:54:13,854 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/.tmp/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/c6571fbaea8e4afe8971633780fcbb4b to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/c6571fbaea8e4afe8971633780fcbb4b 2023-04-19 20:54:13,856 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/.tmp/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/f232dc3763704f0cba7e17c604e3990a to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/C/f232dc3763704f0cba7e17c604e3990a 2023-04-19 20:54:13,859 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/.tmp/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/recovered.edits/930.seqid to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5/recovered.edits/930.seqid 2023-04-19 20:54:13,860 DEBUG [HFileArchiver-3] backup.HFileArchiver(596): Deleted hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/.tmp/data/default/TestAcidGuarantees/f70bfe945d0054ce5a228cf255d902f5 2023-04-19 20:54:13,860 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-19 20:54:13,863 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=15, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-19 20:54:13,883 WARN [PEWorker-5] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-19 20:54:13,925 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-19 20:54:13,926 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=15, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-19 20:54:13,927 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-19 20:54:13,927 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1681937653927"}]},"ts":"9223372036854775807"} 2023-04-19 20:54:13,929 INFO [PEWorker-5] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-19 20:54:13,929 DEBUG [PEWorker-5] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => f70bfe945d0054ce5a228cf255d902f5, NAME => 'TestAcidGuarantees,,1681937620609.f70bfe945d0054ce5a228cf255d902f5.', STARTKEY => '', ENDKEY => ''}] 2023-04-19 20:54:13,929 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-19 20:54:13,929 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1681937653929"}]},"ts":"9223372036854775807"} 2023-04-19 20:54:13,931 INFO [PEWorker-5] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-19 20:54:13,933 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(130): Finished pid=15, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-19 20:54:13,934 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=15, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 111 msec 2023-04-19 20:54:13,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-04-19 20:54:13,935 INFO [Listener at localhost/34893] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 15 completed 2023-04-19 20:54:13,950 INFO [Listener at localhost/34893] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMixedAtomicity Thread=230 (was 206) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-9 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-4 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-5 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Timer for 'HBase' metrics system java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-8 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x2a4b69b2-shared-pool-25 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: (jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-2 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:883) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x2a4b69b2-shared-pool-24 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x2a4b69b2-shared-pool-22 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RSProcedureDispatcher-pool-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-7 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-1-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-717667503_15 at /127.0.0.1:48780 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-10 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-12 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-3 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-6 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS:0;jenkins-hbase4:35247-shortCompactions-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.PriorityBlockingQueue.take(PriorityBlockingQueue.java:549) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x2a4b69b2-shared-pool-23 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-11 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-13 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=545 (was 470) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=396 (was 337) - SystemLoadAverage LEAK? -, ProcessCount=169 (was 169), AvailableMemoryMB=5724 (was 6290) 2023-04-19 20:54:13,961 INFO [Listener at localhost/34893] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobMixedAtomicity Thread=230, OpenFileDescriptor=545, MaxFileDescriptor=60000, SystemLoadAverage=396, ProcessCount=169, AvailableMemoryMB=5724 2023-04-19 20:54:13,963 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-19 20:54:13,963 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-19 20:54:13,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure2.ProcedureExecutor(1029): Stored pid=16, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-19 20:54:13,966 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-19 20:54:13,966 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 16 2023-04-19 20:54:13,966 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-19 20:54:13,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-19 20:54:13,968 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/.tmp/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85 2023-04-19 20:54:13,969 DEBUG [HFileArchiver-4] backup.HFileArchiver(153): Directory hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/.tmp/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85 empty. 2023-04-19 20:54:13,969 DEBUG [HFileArchiver-4] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/.tmp/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85 2023-04-19 20:54:13,969 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-19 20:54:14,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-19 20:54:14,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-19 20:54:14,379 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-19 20:54:14,380 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => c7ee8087efe9c3ae02af767e46864c85, NAME => 'TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/.tmp 2023-04-19 20:54:14,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-19 20:54:14,788 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-19 20:54:14,788 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing c7ee8087efe9c3ae02af767e46864c85, disabling compactions & flushes 2023-04-19 20:54:14,788 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:14,788 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:14,788 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. after waiting 0 ms 2023-04-19 20:54:14,788 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:14,788 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:14,788 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for c7ee8087efe9c3ae02af767e46864c85: 2023-04-19 20:54:14,790 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-19 20:54:14,791 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681937654791"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681937654791"}]},"ts":"1681937654791"} 2023-04-19 20:54:14,793 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-19 20:54:14,794 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-19 20:54:14,794 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681937654794"}]},"ts":"1681937654794"} 2023-04-19 20:54:14,795 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-19 20:54:14,798 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=c7ee8087efe9c3ae02af767e46864c85, ASSIGN}] 2023-04-19 20:54:14,800 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=c7ee8087efe9c3ae02af767e46864c85, ASSIGN 2023-04-19 20:54:14,801 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=c7ee8087efe9c3ae02af767e46864c85, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,35247,1681937616971; forceNewPlan=false, retain=false 2023-04-19 20:54:14,952 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=17 updating hbase:meta row=c7ee8087efe9c3ae02af767e46864c85, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:14,952 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681937654951"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681937654951"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681937654951"}]},"ts":"1681937654951"} 2023-04-19 20:54:14,953 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=18, ppid=17, state=RUNNABLE; OpenRegionProcedure c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971}] 2023-04-19 20:54:15,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-19 20:54:15,109 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:15,109 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => c7ee8087efe9c3ae02af767e46864c85, NAME => 'TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85.', STARTKEY => '', ENDKEY => ''} 2023-04-19 20:54:15,110 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees c7ee8087efe9c3ae02af767e46864c85 2023-04-19 20:54:15,110 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-19 20:54:15,110 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for c7ee8087efe9c3ae02af767e46864c85 2023-04-19 20:54:15,110 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for c7ee8087efe9c3ae02af767e46864c85 2023-04-19 20:54:15,111 INFO [StoreOpener-c7ee8087efe9c3ae02af767e46864c85-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region c7ee8087efe9c3ae02af767e46864c85 2023-04-19 20:54:15,112 DEBUG [StoreOpener-c7ee8087efe9c3ae02af767e46864c85-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A 2023-04-19 20:54:15,112 DEBUG [StoreOpener-c7ee8087efe9c3ae02af767e46864c85-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A 2023-04-19 20:54:15,113 INFO [StoreOpener-c7ee8087efe9c3ae02af767e46864c85-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-19 20:54:15,113 INFO [StoreOpener-c7ee8087efe9c3ae02af767e46864c85-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region c7ee8087efe9c3ae02af767e46864c85 columnFamilyName A 2023-04-19 20:54:15,113 INFO [StoreOpener-c7ee8087efe9c3ae02af767e46864c85-1] regionserver.HStore(310): Store=c7ee8087efe9c3ae02af767e46864c85/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-19 20:54:15,113 INFO [StoreOpener-c7ee8087efe9c3ae02af767e46864c85-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region c7ee8087efe9c3ae02af767e46864c85 2023-04-19 20:54:15,115 DEBUG [StoreOpener-c7ee8087efe9c3ae02af767e46864c85-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B 2023-04-19 20:54:15,115 DEBUG [StoreOpener-c7ee8087efe9c3ae02af767e46864c85-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B 2023-04-19 20:54:15,115 INFO [StoreOpener-c7ee8087efe9c3ae02af767e46864c85-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-19 20:54:15,115 INFO [StoreOpener-c7ee8087efe9c3ae02af767e46864c85-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region c7ee8087efe9c3ae02af767e46864c85 columnFamilyName B 2023-04-19 20:54:15,116 INFO [StoreOpener-c7ee8087efe9c3ae02af767e46864c85-1] regionserver.HStore(310): Store=c7ee8087efe9c3ae02af767e46864c85/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-19 20:54:15,116 INFO [StoreOpener-c7ee8087efe9c3ae02af767e46864c85-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region c7ee8087efe9c3ae02af767e46864c85 2023-04-19 20:54:15,117 DEBUG [StoreOpener-c7ee8087efe9c3ae02af767e46864c85-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C 2023-04-19 20:54:15,117 DEBUG [StoreOpener-c7ee8087efe9c3ae02af767e46864c85-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C 2023-04-19 20:54:15,117 INFO [StoreOpener-c7ee8087efe9c3ae02af767e46864c85-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-19 20:54:15,117 INFO [StoreOpener-c7ee8087efe9c3ae02af767e46864c85-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region c7ee8087efe9c3ae02af767e46864c85 columnFamilyName C 2023-04-19 20:54:15,118 INFO [StoreOpener-c7ee8087efe9c3ae02af767e46864c85-1] regionserver.HStore(310): Store=c7ee8087efe9c3ae02af767e46864c85/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-19 20:54:15,118 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:15,119 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85 2023-04-19 20:54:15,119 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85 2023-04-19 20:54:15,121 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-19 20:54:15,122 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for c7ee8087efe9c3ae02af767e46864c85 2023-04-19 20:54:15,124 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-19 20:54:15,124 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened c7ee8087efe9c3ae02af767e46864c85; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59886242, jitterRate=-0.10762545466423035}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-19 20:54:15,124 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for c7ee8087efe9c3ae02af767e46864c85: 2023-04-19 20:54:15,125 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85., pid=18, masterSystemTime=1681937655105 2023-04-19 20:54:15,126 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:15,126 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:15,127 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=17 updating hbase:meta row=c7ee8087efe9c3ae02af767e46864c85, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:15,127 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681937655127"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681937655127"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681937655127"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681937655127"}]},"ts":"1681937655127"} 2023-04-19 20:54:15,130 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=18, resume processing ppid=17 2023-04-19 20:54:15,130 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=18, ppid=17, state=SUCCESS; OpenRegionProcedure c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 in 175 msec 2023-04-19 20:54:15,131 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=17, resume processing ppid=16 2023-04-19 20:54:15,131 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=17, ppid=16, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=c7ee8087efe9c3ae02af767e46864c85, ASSIGN in 332 msec 2023-04-19 20:54:15,132 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-19 20:54:15,132 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681937655132"}]},"ts":"1681937655132"} 2023-04-19 20:54:15,133 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-19 20:54:15,135 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-19 20:54:15,137 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=16, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1720 sec 2023-04-19 20:54:16,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-19 20:54:16,072 INFO [Listener at localhost/34893] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 16 completed 2023-04-19 20:54:16,073 DEBUG [Listener at localhost/34893] zookeeper.ReadOnlyZKClient(139): Connect 0x237d5550 to 127.0.0.1:58736 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-19 20:54:16,079 DEBUG [Listener at localhost/34893] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2f866a39, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-19 20:54:16,080 DEBUG [Listener at localhost/34893] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-19 20:54:16,084 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56312, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-19 20:54:16,086 DEBUG [Listener at localhost/34893] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-19 20:54:16,088 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53224, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-19 20:54:16,096 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-19 20:54:16,096 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.HMaster$12(2454): Client=jenkins//172.31.14.131 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-19 20:54:16,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure2.ProcedureExecutor(1029): Stored pid=19, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-04-19 20:54:16,519 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-19 20:54:16,520 DEBUG [PEWorker-3] util.FSTableDescriptors(519): Deleted hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-19 20:54:16,520 INFO [PEWorker-3] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-19 20:54:16,524 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=20, ppid=19, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-04-19 20:54:16,537 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=c7ee8087efe9c3ae02af767e46864c85, REOPEN/MOVE}] 2023-04-19 20:54:16,538 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=c7ee8087efe9c3ae02af767e46864c85, REOPEN/MOVE 2023-04-19 20:54:16,538 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=c7ee8087efe9c3ae02af767e46864c85, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:16,538 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681937656538"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681937656538"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681937656538"}]},"ts":"1681937656538"} 2023-04-19 20:54:16,539 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=22, ppid=21, state=RUNNABLE; CloseRegionProcedure c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971}] 2023-04-19 20:54:16,692 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close c7ee8087efe9c3ae02af767e46864c85 2023-04-19 20:54:16,692 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing c7ee8087efe9c3ae02af767e46864c85, disabling compactions & flushes 2023-04-19 20:54:16,693 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:16,693 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:16,693 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. after waiting 0 ms 2023-04-19 20:54:16,693 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:16,696 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-04-19 20:54:16,697 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:16,697 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for c7ee8087efe9c3ae02af767e46864c85: 2023-04-19 20:54:16,697 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(3509): Not adding moved region record: c7ee8087efe9c3ae02af767e46864c85 to self. 2023-04-19 20:54:16,699 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed c7ee8087efe9c3ae02af767e46864c85 2023-04-19 20:54:16,699 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=c7ee8087efe9c3ae02af767e46864c85, regionState=CLOSED 2023-04-19 20:54:16,699 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681937656699"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681937656699"}]},"ts":"1681937656699"} 2023-04-19 20:54:16,702 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=22, resume processing ppid=21 2023-04-19 20:54:16,702 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=22, ppid=21, state=SUCCESS; CloseRegionProcedure c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 in 162 msec 2023-04-19 20:54:16,703 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=c7ee8087efe9c3ae02af767e46864c85, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase4.apache.org,35247,1681937616971; forceNewPlan=false, retain=true 2023-04-19 20:54:16,853 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=c7ee8087efe9c3ae02af767e46864c85, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:16,854 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681937656853"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681937656853"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681937656853"}]},"ts":"1681937656853"} 2023-04-19 20:54:16,856 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=23, ppid=21, state=RUNNABLE; OpenRegionProcedure c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971}] 2023-04-19 20:54:17,011 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:17,011 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => c7ee8087efe9c3ae02af767e46864c85, NAME => 'TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85.', STARTKEY => '', ENDKEY => ''} 2023-04-19 20:54:17,012 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees c7ee8087efe9c3ae02af767e46864c85 2023-04-19 20:54:17,012 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-19 20:54:17,012 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for c7ee8087efe9c3ae02af767e46864c85 2023-04-19 20:54:17,012 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for c7ee8087efe9c3ae02af767e46864c85 2023-04-19 20:54:17,014 INFO [StoreOpener-c7ee8087efe9c3ae02af767e46864c85-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region c7ee8087efe9c3ae02af767e46864c85 2023-04-19 20:54:17,015 DEBUG [StoreOpener-c7ee8087efe9c3ae02af767e46864c85-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A 2023-04-19 20:54:17,015 DEBUG [StoreOpener-c7ee8087efe9c3ae02af767e46864c85-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A 2023-04-19 20:54:17,015 INFO [StoreOpener-c7ee8087efe9c3ae02af767e46864c85-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-19 20:54:17,018 INFO [StoreOpener-c7ee8087efe9c3ae02af767e46864c85-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region c7ee8087efe9c3ae02af767e46864c85 columnFamilyName A 2023-04-19 20:54:17,020 INFO [StoreOpener-c7ee8087efe9c3ae02af767e46864c85-1] regionserver.HStore(310): Store=c7ee8087efe9c3ae02af767e46864c85/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-19 20:54:17,020 INFO [StoreOpener-c7ee8087efe9c3ae02af767e46864c85-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region c7ee8087efe9c3ae02af767e46864c85 2023-04-19 20:54:17,021 DEBUG [StoreOpener-c7ee8087efe9c3ae02af767e46864c85-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B 2023-04-19 20:54:17,021 DEBUG [StoreOpener-c7ee8087efe9c3ae02af767e46864c85-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B 2023-04-19 20:54:17,022 INFO [StoreOpener-c7ee8087efe9c3ae02af767e46864c85-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-19 20:54:17,022 INFO [StoreOpener-c7ee8087efe9c3ae02af767e46864c85-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region c7ee8087efe9c3ae02af767e46864c85 columnFamilyName B 2023-04-19 20:54:17,022 INFO [StoreOpener-c7ee8087efe9c3ae02af767e46864c85-1] regionserver.HStore(310): Store=c7ee8087efe9c3ae02af767e46864c85/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-19 20:54:17,023 INFO [StoreOpener-c7ee8087efe9c3ae02af767e46864c85-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region c7ee8087efe9c3ae02af767e46864c85 2023-04-19 20:54:17,023 DEBUG [StoreOpener-c7ee8087efe9c3ae02af767e46864c85-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C 2023-04-19 20:54:17,023 DEBUG [StoreOpener-c7ee8087efe9c3ae02af767e46864c85-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C 2023-04-19 20:54:17,024 INFO [StoreOpener-c7ee8087efe9c3ae02af767e46864c85-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-19 20:54:17,024 INFO [StoreOpener-c7ee8087efe9c3ae02af767e46864c85-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region c7ee8087efe9c3ae02af767e46864c85 columnFamilyName C 2023-04-19 20:54:17,025 INFO [StoreOpener-c7ee8087efe9c3ae02af767e46864c85-1] regionserver.HStore(310): Store=c7ee8087efe9c3ae02af767e46864c85/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-19 20:54:17,025 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:17,025 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85 2023-04-19 20:54:17,026 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85 2023-04-19 20:54:17,028 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-19 20:54:17,029 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for c7ee8087efe9c3ae02af767e46864c85 2023-04-19 20:54:17,030 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened c7ee8087efe9c3ae02af767e46864c85; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=67543418, jitterRate=0.006475359201431274}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-19 20:54:17,030 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for c7ee8087efe9c3ae02af767e46864c85: 2023-04-19 20:54:17,031 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85., pid=23, masterSystemTime=1681937657007 2023-04-19 20:54:17,032 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:17,032 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:17,033 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=c7ee8087efe9c3ae02af767e46864c85, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:17,033 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681937657033"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681937657033"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681937657033"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681937657033"}]},"ts":"1681937657033"} 2023-04-19 20:54:17,036 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=23, resume processing ppid=21 2023-04-19 20:54:17,036 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=23, ppid=21, state=SUCCESS; OpenRegionProcedure c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 in 179 msec 2023-04-19 20:54:17,037 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=21, resume processing ppid=20 2023-04-19 20:54:17,038 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=21, ppid=20, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=c7ee8087efe9c3ae02af767e46864c85, REOPEN/MOVE in 499 msec 2023-04-19 20:54:17,041 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=20, resume processing ppid=19 2023-04-19 20:54:17,041 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=20, ppid=19, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 516 msec 2023-04-19 20:54:17,043 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=19, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 942 msec 2023-04-19 20:54:17,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1227): Checking to see if procedure is done pid=19 2023-04-19 20:54:17,053 DEBUG [Listener at localhost/34893] zookeeper.ReadOnlyZKClient(139): Connect 0x5c508332 to 127.0.0.1:58736 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-19 20:54:17,059 DEBUG [Listener at localhost/34893] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@63ebf1e9, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-19 20:54:17,060 DEBUG [Listener at localhost/34893] zookeeper.ReadOnlyZKClient(139): Connect 0x16323966 to 127.0.0.1:58736 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-19 20:54:17,065 DEBUG [Listener at localhost/34893] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@28fcbe97, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-19 20:54:17,066 DEBUG [Listener at localhost/34893] zookeeper.ReadOnlyZKClient(139): Connect 0x52f29016 to 127.0.0.1:58736 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-19 20:54:17,069 DEBUG [Listener at localhost/34893] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@550c7a27, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-19 20:54:17,070 DEBUG [Listener at localhost/34893] zookeeper.ReadOnlyZKClient(139): Connect 0x40c57efc to 127.0.0.1:58736 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-19 20:54:17,074 DEBUG [Listener at localhost/34893] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@76aef981, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-19 20:54:17,074 DEBUG [Listener at localhost/34893] zookeeper.ReadOnlyZKClient(139): Connect 0x2a5a81b5 to 127.0.0.1:58736 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-19 20:54:17,078 DEBUG [Listener at localhost/34893] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4183389c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-19 20:54:17,078 DEBUG [Listener at localhost/34893] zookeeper.ReadOnlyZKClient(139): Connect 0x3bccfdc5 to 127.0.0.1:58736 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-19 20:54:17,083 DEBUG [Listener at localhost/34893] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@63d375f8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-19 20:54:17,083 DEBUG [Listener at localhost/34893] zookeeper.ReadOnlyZKClient(139): Connect 0x4fbb4986 to 127.0.0.1:58736 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-19 20:54:17,087 DEBUG [Listener at localhost/34893] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1345b761, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-19 20:54:17,087 DEBUG [Listener at localhost/34893] zookeeper.ReadOnlyZKClient(139): Connect 0x0b69850d to 127.0.0.1:58736 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-19 20:54:17,090 DEBUG [Listener at localhost/34893] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1e0b0a9f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-19 20:54:17,091 DEBUG [Listener at localhost/34893] zookeeper.ReadOnlyZKClient(139): Connect 0x40883381 to 127.0.0.1:58736 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-19 20:54:17,094 DEBUG [Listener at localhost/34893] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@78e8770, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-19 20:54:17,104 DEBUG [hconnection-0x72931772-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-19 20:54:17,104 DEBUG [hconnection-0x5db39bf7-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-19 20:54:17,104 DEBUG [hconnection-0x28e2a3c0-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-19 20:54:17,105 DEBUG [hconnection-0x6e23d3b8-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-19 20:54:17,105 DEBUG [hconnection-0x1e37b54f-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-19 20:54:17,108 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56352, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-19 20:54:17,111 DEBUG [hconnection-0x6fc0b884-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-19 20:54:17,111 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56326, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-19 20:54:17,111 DEBUG [hconnection-0x6a5684ce-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-19 20:54:17,113 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56366, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-19 20:54:17,113 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56328, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-19 20:54:17,112 DEBUG [hconnection-0x22138d87-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-19 20:54:17,113 DEBUG [hconnection-0x3e56a364-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-19 20:54:17,116 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56368, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-19 20:54:17,116 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56384, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-19 20:54:17,118 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56408, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-19 20:54:17,119 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56392, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-19 20:54:17,124 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56398, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-19 20:54:17,130 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:54:17,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(9158): Flush requested on c7ee8087efe9c3ae02af767e46864c85 2023-04-19 20:54:17,141 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c7ee8087efe9c3ae02af767e46864c85 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-19 20:54:17,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:54:17,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:54:17,141 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:54:17,141 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:17,141 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:54:17,141 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:54:17,142 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:17,142 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:17,142 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c7ee8087efe9c3ae02af767e46864c85, store=A 2023-04-19 20:54:17,143 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:54:17,143 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c7ee8087efe9c3ae02af767e46864c85, store=B 2023-04-19 20:54:17,143 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:54:17,143 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c7ee8087efe9c3ae02af767e46864c85, store=C 2023-04-19 20:54:17,143 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:54:17,148 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:17,149 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:17,149 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:54:17,149 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:17,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:17,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:17,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:17,151 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:17,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:54:17,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:17,152 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:54:17,162 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:54:17,162 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:54:17,162 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:54:17,162 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:17,162 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:54:17,168 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:54:17,168 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:54:17,168 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:17,168 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. started... 2023-04-19 20:54:17,168 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. as already flushing 2023-04-19 20:54:17,168 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:17,168 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:54:17,168 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:54:17,168 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:17,172 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:54:17,173 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:54:17,173 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:17,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56326 deadline: 1681937717163, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:17,174 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:17,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56368 deadline: 1681937717163, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:17,174 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:17,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56408 deadline: 1681937717163, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:17,175 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:17,175 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:17,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:17,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:17,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:17,175 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:17,176 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:54:17,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:17,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:17,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:17,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:17,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:17,177 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:54:17,178 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3e1cb703[Count = 0] remaining members to acquire global barrier 2023-04-19 20:54:17,178 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:54:17,178 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:17,179 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:17,179 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:17,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:17,179 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:54:17,179 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:54:17,179 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:54:17,179 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:17,179 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:54:17,179 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:17,180 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:17,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56384 deadline: 1681937717179, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:17,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56352 deadline: 1681937717179, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:17,186 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:17,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:17,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:17,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:17,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:17,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:17,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:17,187 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:54:17,187 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:17,187 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:54:17,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:17,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:17,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:17,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:17,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:54:17,189 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:54:17,189 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:54:17,189 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:54:17,189 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:54:17,189 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:54:17,189 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:54:17,191 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:17,191 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:17,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:17,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:17,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:17,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:17,191 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:17,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:17,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:17,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:17,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:17,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:17,193 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:17,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:17,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:17,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:17,194 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:17,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:17,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:17,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:17,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:17,198 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:17,198 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:17,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:17,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:17,199 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:17,199 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:17,199 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:17,199 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:54:17,199 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:54:17,199 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:17,199 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:17,199 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:17,199 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:17,199 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:17,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:17,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:17,199 DEBUG [Thread-837] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:54:17,199 DEBUG [Thread-837] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:54:17,199 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:54:17,207 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304197f8ec5398e17499d9da3fed46b2afdd0 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304197f8ec5398e17499d9da3fed46b2afdd0 2023-04-19 20:54:17,239 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=18, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/A/03bec7be52ca4b7c9d6044c713757b35 2023-04-19 20:54:17,265 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=18 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/B/0aba3f28101349eb99fb5dbb80cdfd55 2023-04-19 20:54:17,278 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:17,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56326 deadline: 1681937717277, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:17,281 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:17,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56408 deadline: 1681937717280, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:17,282 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:17,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56368 deadline: 1681937717280, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:17,282 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:17,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56384 deadline: 1681937717281, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:17,282 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:17,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56352 deadline: 1681937717281, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:17,310 DEBUG [Thread-837] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:54:17,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:54:17,313 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:54:17,316 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=18 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/C/7334fe85469e4371908bd37f2ae522e1 2023-04-19 20:54:17,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:54:17,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:54:17,319 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:54:17,319 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:17,319 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:54:17,320 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:54:17,320 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:17,320 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:17,321 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:17,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:17,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:17,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:17,322 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:17,323 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:17,323 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:54:17,323 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:17,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:54:17,324 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/A/03bec7be52ca4b7c9d6044c713757b35 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/03bec7be52ca4b7c9d6044c713757b35 2023-04-19 20:54:17,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:17,324 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:54:17,330 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/03bec7be52ca4b7c9d6044c713757b35, entries=150, sequenceid=18, filesize=25.1 K 2023-04-19 20:54:17,331 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/B/0aba3f28101349eb99fb5dbb80cdfd55 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/0aba3f28101349eb99fb5dbb80cdfd55 2023-04-19 20:54:17,336 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:54:17,336 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:54:17,336 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:17,336 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:54:17,341 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:54:17,341 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:54:17,341 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:17,344 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. started... 2023-04-19 20:54:17,344 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. as already flushing 2023-04-19 20:54:17,344 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:17,344 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:54:17,344 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:54:17,344 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:17,345 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:54:17,345 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:54:17,350 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/0aba3f28101349eb99fb5dbb80cdfd55, entries=150, sequenceid=18, filesize=11.6 K 2023-04-19 20:54:17,351 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:17,351 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:17,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:17,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:17,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:17,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:17,351 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:17,351 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:54:17,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:17,352 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/C/7334fe85469e4371908bd37f2ae522e1 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/7334fe85469e4371908bd37f2ae522e1 2023-04-19 20:54:17,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:17,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:17,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:17,353 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:54:17,353 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@79482e33[Count = 0] remaining members to acquire global barrier 2023-04-19 20:54:17,353 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:54:17,354 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:17,355 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:17,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:17,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:17,355 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:54:17,355 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:54:17,355 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:54:17,356 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:17,356 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:54:17,357 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:17,357 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:54:17,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:17,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:17,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:17,357 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:17,357 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:54:17,357 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/7334fe85469e4371908bd37f2ae522e1, entries=150, sequenceid=18, filesize=11.6 K 2023-04-19 20:54:17,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:17,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:17,358 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for c7ee8087efe9c3ae02af767e46864c85 in 217ms, sequenceid=18, compaction requested=false 2023-04-19 20:54:17,358 DEBUG [MemStoreFlusher.0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-04-19 20:54:17,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:17,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:17,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:17,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:17,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:17,360 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c7ee8087efe9c3ae02af767e46864c85: 2023-04-19 20:54:17,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:54:17,365 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:54:17,366 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:54:17,366 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:54:17,366 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:54:17,366 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:54:17,366 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:54:17,367 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:17,368 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:17,368 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:17,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:17,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:17,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:17,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:17,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:17,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:17,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:17,369 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:17,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:17,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:17,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:17,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:17,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:17,370 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:17,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:17,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:17,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:17,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:17,374 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:17,374 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:17,374 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:17,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:17,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:17,374 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:17,374 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:17,374 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:17,374 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:17,374 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:17,375 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:17,375 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:17,374 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:54:17,375 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:54:17,375 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:54:17,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:17,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:17,375 DEBUG [Thread-837] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:54:17,375 DEBUG [Thread-837] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:54:17,475 DEBUG [Thread-837] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:54:17,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:54:17,478 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:54:17,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(9158): Flush requested on c7ee8087efe9c3ae02af767e46864c85 2023-04-19 20:54:17,483 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c7ee8087efe9c3ae02af767e46864c85 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-19 20:54:17,483 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c7ee8087efe9c3ae02af767e46864c85, store=A 2023-04-19 20:54:17,483 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:54:17,483 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c7ee8087efe9c3ae02af767e46864c85, store=B 2023-04-19 20:54:17,483 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:54:17,483 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c7ee8087efe9c3ae02af767e46864c85, store=C 2023-04-19 20:54:17,483 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:54:17,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:54:17,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:54:17,489 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:54:17,489 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:17,490 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:54:17,490 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:54:17,490 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:17,490 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:17,492 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:17,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:17,492 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:17,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:17,492 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:17,492 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:54:17,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:17,493 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:17,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:54:17,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:17,493 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:54:17,500 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:17,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56326 deadline: 1681937717499, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:17,501 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:17,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56408 deadline: 1681937717500, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:17,503 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:17,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56368 deadline: 1681937717500, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:17,504 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:54:17,504 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:54:17,504 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:54:17,504 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:17,504 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:54:17,510 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:17,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56352 deadline: 1681937717510, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:17,510 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:17,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56384 deadline: 1681937717510, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:17,512 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:54:17,512 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:54:17,512 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:17,512 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. started... 2023-04-19 20:54:17,512 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. as already flushing 2023-04-19 20:54:17,512 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:17,512 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:54:17,512 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:54:17,513 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:17,518 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:54:17,519 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:54:17,521 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:17,521 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:17,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:17,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:17,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:17,521 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:17,521 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:54:17,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:17,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:17,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:17,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:17,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:17,526 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:54:17,526 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@47effd91[Count = 0] remaining members to acquire global barrier 2023-04-19 20:54:17,526 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:54:17,526 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:17,528 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:17,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:17,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:17,528 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:54:17,528 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:54:17,528 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:54:17,528 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:17,528 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:54:17,531 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:17,531 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:54:17,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:17,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:17,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:17,531 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:17,532 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:54:17,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:17,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:17,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:17,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:17,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:17,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:17,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:17,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:54:17,535 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:54:17,535 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:54:17,535 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:54:17,535 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:54:17,535 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:54:17,535 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:54:17,537 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:17,537 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:17,537 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:17,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:17,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:17,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:17,537 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:17,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:17,537 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:17,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:17,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:17,538 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:17,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:17,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:17,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:17,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:17,539 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:17,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:17,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:17,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:17,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:17,542 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:17,542 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:17,543 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:17,543 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:17,543 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:17,543 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:17,543 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:17,543 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:17,543 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:17,543 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:54:17,543 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:54:17,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:17,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:17,543 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:54:17,543 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:17,543 DEBUG [Thread-837] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:54:17,543 DEBUG [Thread-837] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:54:17,544 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:17,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:17,603 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:17,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56326 deadline: 1681937717602, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:17,603 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:17,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56408 deadline: 1681937717602, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:17,605 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:17,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56368 deadline: 1681937717604, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:17,643 DEBUG [Thread-837] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:54:17,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:54:17,650 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:54:17,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:54:17,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:54:17,656 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:54:17,656 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:17,656 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:54:17,656 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:54:17,657 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:17,657 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:17,658 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:17,658 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:17,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:17,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:17,658 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:17,658 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:54:17,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:17,659 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:17,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:54:17,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:17,660 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:54:17,664 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:54:17,664 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:54:17,664 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:54:17,664 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:17,664 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:54:17,668 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:54:17,668 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:54:17,668 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:17,668 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. started... 2023-04-19 20:54:17,668 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. as already flushing 2023-04-19 20:54:17,668 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:17,668 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:54:17,668 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:54:17,668 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:17,672 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:54:17,672 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:54:17,677 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:17,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:17,677 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:17,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:17,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:17,678 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:17,678 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:54:17,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:17,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:17,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:17,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:17,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:17,679 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:54:17,679 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@16636984[Count = 0] remaining members to acquire global barrier 2023-04-19 20:54:17,679 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:54:17,679 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:17,681 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:17,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:17,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:17,681 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:54:17,681 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:54:17,682 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:54:17,682 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:17,682 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:54:17,684 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:54:17,684 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:17,684 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:17,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:17,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:17,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:17,684 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:54:17,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:17,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:17,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:17,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:17,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:17,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:17,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:17,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:54:17,687 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:54:17,687 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:54:17,687 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:54:17,687 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:54:17,687 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:54:17,687 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:54:17,690 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:17,690 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:17,690 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:17,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:17,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:17,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:17,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:17,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:17,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:17,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:17,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:17,691 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:17,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:17,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:17,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:17,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:17,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:17,693 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:17,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:17,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:17,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:17,695 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:17,695 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:17,696 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:17,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:17,696 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:17,696 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:54:17,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:17,696 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:54:17,696 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:54:17,696 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:17,696 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:17,696 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:17,696 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:17,696 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:17,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:17,696 DEBUG [Thread-837] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:54:17,696 DEBUG [Thread-837] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:54:17,696 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:17,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:17,796 DEBUG [Thread-837] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:54:17,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:54:17,799 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:54:17,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:54:17,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:54:17,803 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:54:17,803 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:17,803 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:54:17,803 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:54:17,804 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:17,804 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:17,805 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:17,805 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:17,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56326 deadline: 1681937717805, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:17,806 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:17,806 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:54:17,806 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:17,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:17,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:17,806 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:17,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56408 deadline: 1681937717805, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:17,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:17,807 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:17,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:54:17,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:17,807 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:54:17,807 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:54:17,808 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:54:17,808 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:54:17,808 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:17,808 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:54:17,808 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:17,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56368 deadline: 1681937717806, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:17,812 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:54:17,812 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:54:17,812 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:17,812 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. started... 2023-04-19 20:54:17,812 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. as already flushing 2023-04-19 20:54:17,812 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:17,812 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:54:17,812 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:54:17,812 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:17,813 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:54:17,813 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:54:17,814 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:17,814 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:17,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56352 deadline: 1681937717812, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:17,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56384 deadline: 1681937717812, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:17,814 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:17,814 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:17,814 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:17,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:17,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:17,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:17,814 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:54:17,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:17,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:17,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:17,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:17,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:17,816 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:54:17,816 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d1c60e4[Count = 0] remaining members to acquire global barrier 2023-04-19 20:54:17,816 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:54:17,816 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:17,819 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:17,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:17,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:17,819 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:54:17,819 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:54:17,819 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:54:17,819 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:17,819 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:54:17,820 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:17,820 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:54:17,821 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:17,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:17,821 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:54:17,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:17,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:17,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:17,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:17,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:17,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:17,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:17,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:17,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:17,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:54:17,824 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:54:17,824 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:54:17,824 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:54:17,824 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:54:17,824 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:54:17,824 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:54:17,826 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:17,826 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:17,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:17,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:17,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:17,826 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:17,826 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:17,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:17,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:17,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:17,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:17,827 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:17,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:17,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:17,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:17,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:17,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:17,828 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:17,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:17,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:17,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:17,830 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:17,830 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:17,830 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:17,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:17,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:17,830 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:54:17,830 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:54:17,831 DEBUG [Thread-837] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:54:17,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:17,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:17,831 DEBUG [Thread-837] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:54:17,831 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:17,831 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:17,831 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:17,831 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:17,831 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:17,831 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:17,831 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:17,931 DEBUG [Thread-837] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:54:17,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:54:17,932 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230419087340cf731f4534b45c4cfaad10fb96 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419087340cf731f4534b45c4cfaad10fb96 2023-04-19 20:54:17,934 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:54:17,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:54:17,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:54:17,937 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:54:17,937 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:17,937 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:54:17,937 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:54:17,937 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:17,937 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:17,940 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:17,940 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:17,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:17,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:17,940 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:17,940 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:54:17,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:17,941 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:17,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:54:17,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:17,941 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:54:17,942 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:54:17,942 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:54:17,942 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:54:17,942 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:17,942 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:54:17,944 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:54:17,944 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:54:17,944 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:17,944 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. started... 2023-04-19 20:54:17,944 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. as already flushing 2023-04-19 20:54:17,944 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:17,944 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:54:17,944 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:54:17,944 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:17,944 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:54:17,944 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:54:17,946 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=37, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/A/101e02bda7a447a5ba7839a7ea2044cf 2023-04-19 20:54:17,946 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:17,946 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:17,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:17,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:17,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:17,946 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:17,946 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:54:17,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:17,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:17,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:17,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:17,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:17,948 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:54:17,948 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@759df6c2[Count = 0] remaining members to acquire global barrier 2023-04-19 20:54:17,948 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:54:17,948 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:17,949 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:17,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:17,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:17,949 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:54:17,949 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:54:17,949 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:54:17,950 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:17,950 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:54:17,951 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:17,951 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:54:17,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:17,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:17,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:17,951 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:17,951 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:54:17,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:17,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:17,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:17,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:17,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:17,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:17,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:17,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:54:17,954 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:54:17,954 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:54:17,955 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:54:17,955 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:54:17,955 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:54:17,955 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:54:17,958 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:17,958 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:17,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:17,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:17,959 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:17,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:17,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:17,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:17,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:17,959 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:17,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:17,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:17,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:17,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:17,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:17,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:17,961 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:17,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:17,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:17,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:17,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:17,963 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:17,963 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:17,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:17,963 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:17,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:17,963 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:17,963 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:54:17,963 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:54:17,963 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:17,963 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:17,963 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:17,963 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:17,963 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:17,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:17,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:17,964 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:54:17,963 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:17,964 DEBUG [Thread-837] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:54:17,964 DEBUG [Thread-837] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:54:18,064 DEBUG [Thread-837] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:54:18,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:54:18,067 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:54:18,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:54:18,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:54:18,068 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:54:18,069 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:18,069 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:54:18,069 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:54:18,069 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:18,069 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:18,071 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:18,071 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:18,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:18,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:18,071 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:18,072 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:54:18,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:18,072 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:18,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:54:18,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:18,072 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:54:18,073 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:54:18,073 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:54:18,073 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:54:18,073 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:18,073 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:54:18,073 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:54:18,073 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:54:18,073 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:18,073 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. started... 2023-04-19 20:54:18,073 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. as already flushing 2023-04-19 20:54:18,073 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:18,073 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:54:18,073 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:54:18,073 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:18,073 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:54:18,073 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:54:18,075 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:18,075 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:18,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:18,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:18,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:18,075 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:18,075 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:54:18,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:18,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:18,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:18,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:18,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:18,077 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:54:18,077 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4da68e95[Count = 0] remaining members to acquire global barrier 2023-04-19 20:54:18,077 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:54:18,077 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:18,084 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:18,084 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:18,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:18,084 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:54:18,084 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:54:18,084 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:54:18,084 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:18,085 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:54:18,088 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:18,088 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:54:18,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:18,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:18,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:18,088 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:18,089 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:54:18,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:18,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:18,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:18,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:18,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:18,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:18,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:18,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:54:18,091 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:54:18,091 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:54:18,091 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:54:18,091 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:54:18,091 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:54:18,091 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:54:18,093 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:18,093 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:18,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:18,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:18,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:18,093 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:18,093 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:18,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:18,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:18,093 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:18,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:18,093 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:18,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:18,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:18,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:18,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:18,094 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:18,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:18,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:18,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:18,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:18,097 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:18,097 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:18,097 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:54:18,097 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:54:18,097 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:18,097 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:54:18,097 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:18,097 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:18,097 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:18,097 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:18,097 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:18,097 DEBUG [Thread-837] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:54:18,097 DEBUG [Thread-837] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:54:18,097 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:18,097 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:18,097 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:18,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:18,098 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:18,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:18,107 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:18,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56326 deadline: 1681937718107, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:18,108 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:18,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56408 deadline: 1681937718107, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:18,109 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:18,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56368 deadline: 1681937718109, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:18,197 DEBUG [Thread-837] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:54:18,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:54:18,199 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:54:18,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:54:18,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:54:18,201 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:54:18,201 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:18,201 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:54:18,201 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:54:18,201 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:18,201 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:18,208 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:18,208 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:18,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:18,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:18,209 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:18,209 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:54:18,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:18,209 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:18,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:54:18,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:18,210 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:54:18,210 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:54:18,210 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:54:18,210 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:54:18,210 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:18,210 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:54:18,210 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:54:18,210 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:54:18,210 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:18,210 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. started... 2023-04-19 20:54:18,211 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. as already flushing 2023-04-19 20:54:18,211 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:18,211 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:54:18,211 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:54:18,211 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:18,211 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:54:18,211 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:54:18,213 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:18,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:18,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:18,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:18,213 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:18,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:18,213 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:18,213 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:54:18,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:18,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:18,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:18,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:18,215 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:54:18,215 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4084e1ec[Count = 0] remaining members to acquire global barrier 2023-04-19 20:54:18,215 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:54:18,215 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:18,216 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:18,216 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:18,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:18,216 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:54:18,216 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:54:18,216 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:18,216 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:54:18,216 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:54:18,218 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:18,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:18,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:18,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:18,218 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:54:18,218 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:18,218 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:54:18,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:18,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:18,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:18,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:18,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:18,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:18,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:18,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:54:18,221 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:54:18,221 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:54:18,221 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:54:18,221 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:54:18,221 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:54:18,221 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:54:18,224 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:18,224 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:18,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:18,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:18,224 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:18,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:18,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:18,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:18,225 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:18,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:18,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:18,225 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:18,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:18,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:18,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:18,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:18,226 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:18,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:18,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:18,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:18,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:18,229 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:18,229 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:18,229 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:18,229 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:18,229 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:18,229 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:18,229 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:18,229 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:54:18,230 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:54:18,229 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:18,230 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:18,230 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:18,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:18,230 DEBUG [Thread-837] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:54:18,230 DEBUG [Thread-837] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:54:18,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:18,230 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:54:18,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:18,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:18,323 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:18,323 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:18,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56352 deadline: 1681937718320, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:18,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56384 deadline: 1681937718320, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:18,330 DEBUG [Thread-837] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:54:18,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:54:18,335 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:54:18,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:54:18,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:54:18,341 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:54:18,342 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:18,342 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:54:18,342 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:54:18,342 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:18,342 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:18,343 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:18,344 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:18,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:18,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:18,344 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:18,344 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:54:18,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:18,345 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:18,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:54:18,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:18,345 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:54:18,347 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:54:18,347 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:54:18,347 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:54:18,348 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:18,348 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:54:18,351 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:54:18,353 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:54:18,353 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:18,356 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. started... 2023-04-19 20:54:18,356 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. as already flushing 2023-04-19 20:54:18,356 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:18,356 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:54:18,356 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:54:18,356 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:18,356 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:54:18,356 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:54:18,358 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:18,358 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:18,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:18,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:18,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:18,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:18,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:18,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:18,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:18,360 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:18,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:18,360 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:54:18,361 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:54:18,361 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6eb1b258[Count = 0] remaining members to acquire global barrier 2023-04-19 20:54:18,361 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:54:18,361 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:18,363 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=37 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/B/9bdaa4f67193473b9d899f5802a4eb47 2023-04-19 20:54:18,367 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:18,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:18,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:18,367 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:54:18,367 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:54:18,367 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:54:18,367 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:18,367 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:54:18,370 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:54:18,370 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:18,370 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:54:18,371 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:18,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:18,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:18,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:18,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:18,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:18,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:18,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:18,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:18,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:18,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:18,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:54:18,375 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:54:18,375 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:54:18,375 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:54:18,375 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:54:18,375 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:54:18,375 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:54:18,377 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:18,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:18,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:18,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:18,377 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:18,378 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:18,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:18,378 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:18,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:18,378 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:18,378 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:18,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:18,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:18,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:18,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:18,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:18,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:18,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:18,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:18,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:18,380 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:18,383 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:18,383 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:18,383 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:18,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:18,383 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:54:18,383 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:54:18,383 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:18,383 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:18,383 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:18,383 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:18,383 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:18,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:18,383 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:18,383 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:18,383 DEBUG [Thread-837] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:54:18,383 DEBUG [Thread-837] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:54:18,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:18,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:18,483 DEBUG [Thread-837] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:54:18,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:54:18,486 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:54:18,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:54:18,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:54:18,488 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:54:18,488 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:18,488 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:54:18,488 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:54:18,488 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:18,488 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:18,491 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:18,491 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:18,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:18,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:18,491 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:18,491 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:54:18,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:18,492 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:18,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:54:18,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:18,492 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:54:18,493 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:54:18,493 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:54:18,493 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:54:18,493 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:18,493 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:54:18,498 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:54:18,498 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:54:18,498 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:18,498 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. started... 2023-04-19 20:54:18,498 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. as already flushing 2023-04-19 20:54:18,498 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:18,498 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:54:18,498 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:54:18,498 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:18,498 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:54:18,498 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:54:18,500 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:18,500 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:18,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:18,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:18,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:18,500 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:18,500 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:54:18,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:18,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:18,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:18,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:18,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:18,502 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:54:18,502 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@740072aa[Count = 0] remaining members to acquire global barrier 2023-04-19 20:54:18,502 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:54:18,502 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:18,504 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:18,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:18,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:18,504 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:54:18,504 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:54:18,504 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:54:18,504 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:18,504 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:54:18,505 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:18,505 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:54:18,506 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:18,506 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:54:18,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:18,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:18,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:18,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:18,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:18,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:18,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:18,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:18,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:18,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:18,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:54:18,510 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:54:18,510 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:54:18,510 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:54:18,510 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:54:18,510 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:54:18,510 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:54:18,519 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:18,520 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:18,520 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:18,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:18,520 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:18,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:18,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:18,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:18,520 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:18,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:18,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:18,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:18,521 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:18,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:18,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:18,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:18,522 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:18,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:18,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:18,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:18,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:18,531 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:18,532 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:18,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:18,532 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:18,532 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:18,532 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:18,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:18,532 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:54:18,532 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:54:18,532 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:18,532 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:18,532 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:18,532 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:18,532 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:18,532 DEBUG [Thread-837] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:54:18,532 DEBUG [Thread-837] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:54:18,532 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:54:18,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:18,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:18,613 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:18,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56326 deadline: 1681937718612, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:18,613 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:18,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56408 deadline: 1681937718612, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:18,614 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:18,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56368 deadline: 1681937718614, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:18,632 DEBUG [Thread-837] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:54:18,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:54:18,635 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:54:18,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:54:18,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:54:18,636 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:54:18,636 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:18,636 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:54:18,636 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:54:18,637 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:18,637 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:18,638 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:18,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:18,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:18,638 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:18,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:18,639 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:18,639 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:54:18,639 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:18,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:54:18,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:18,639 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:54:18,639 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:54:18,639 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:54:18,639 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:54:18,639 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:18,640 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:54:18,640 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:54:18,640 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:54:18,640 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:18,640 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. started... 2023-04-19 20:54:18,640 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. as already flushing 2023-04-19 20:54:18,640 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:18,640 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:54:18,640 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:54:18,640 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:18,640 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:54:18,640 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:54:18,643 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:18,643 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:18,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:18,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:18,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:18,643 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:18,643 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:54:18,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:18,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:18,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:18,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:18,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:18,645 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:54:18,645 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@dab9899[Count = 0] remaining members to acquire global barrier 2023-04-19 20:54:18,645 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:54:18,645 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:18,646 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:18,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:18,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:18,646 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:54:18,646 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:54:18,646 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:54:18,647 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:18,647 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:54:18,648 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:18,648 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:54:18,648 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:18,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:18,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:18,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:18,648 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:54:18,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:18,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:18,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:18,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:18,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:18,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:18,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:18,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:54:18,651 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:54:18,651 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:54:18,651 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:54:18,651 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:54:18,651 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:54:18,651 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:54:18,653 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:18,653 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:18,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:18,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:18,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:18,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:18,653 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:18,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:18,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:18,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:18,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:18,653 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:18,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:18,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:18,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:18,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:18,655 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:18,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:18,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:18,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:18,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:18,658 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:18,658 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:18,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:18,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:18,658 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:54:18,658 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:54:18,658 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:18,658 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:54:18,658 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:18,658 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:18,658 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:18,658 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:18,658 DEBUG [Thread-837] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:54:18,658 DEBUG [Thread-837] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:54:18,658 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:18,658 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:18,659 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:18,659 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:18,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:18,759 DEBUG [Thread-837] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:54:18,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:54:18,761 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:54:18,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:54:18,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:54:18,762 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:54:18,762 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:18,762 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:54:18,763 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:54:18,763 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:18,763 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:18,764 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:18,764 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:18,764 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:18,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:18,765 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:18,765 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:54:18,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:18,765 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:18,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:54:18,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:18,766 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:54:18,766 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:54:18,766 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:54:18,766 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:54:18,766 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:18,766 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:54:18,766 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:54:18,767 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:54:18,767 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:18,767 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. started... 2023-04-19 20:54:18,767 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. as already flushing 2023-04-19 20:54:18,767 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:18,767 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:54:18,767 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:54:18,767 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:18,767 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:54:18,767 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:54:18,770 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:18,770 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:18,770 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:54:18,772 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:18,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:18,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:18,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:18,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:18,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:18,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:18,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:18,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:18,774 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:54:18,774 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a478341[Count = 0] remaining members to acquire global barrier 2023-04-19 20:54:18,774 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:54:18,774 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:18,775 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:18,775 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:18,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:18,775 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:54:18,775 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:54:18,775 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:54:18,775 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:18,775 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:54:18,777 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:54:18,777 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:18,777 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:18,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:18,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:18,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:18,777 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:54:18,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:18,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:18,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:18,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:18,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:18,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:18,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:18,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:54:18,780 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:54:18,780 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:54:18,780 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:54:18,780 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:54:18,780 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:54:18,780 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:54:18,781 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:18,782 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:18,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:18,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:18,782 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:18,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:18,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:18,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:18,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:18,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:18,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:18,783 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:18,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:18,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:18,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:18,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:18,786 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:18,786 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=37 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/C/69aa733f9b70435ab3b80dfd86dcdbba 2023-04-19 20:54:18,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:18,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:18,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:18,790 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:18,790 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:18,790 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:18,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:18,790 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:18,791 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:18,791 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:18,791 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:54:18,791 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:54:18,791 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:18,791 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:18,791 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:18,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:18,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:18,791 DEBUG [Thread-837] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:54:18,791 DEBUG [Thread-837] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:54:18,791 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:18,791 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:18,792 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:54:18,792 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/A/101e02bda7a447a5ba7839a7ea2044cf as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/101e02bda7a447a5ba7839a7ea2044cf 2023-04-19 20:54:18,798 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/101e02bda7a447a5ba7839a7ea2044cf, entries=200, sequenceid=37, filesize=31.9 K 2023-04-19 20:54:18,807 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/B/9bdaa4f67193473b9d899f5802a4eb47 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/9bdaa4f67193473b9d899f5802a4eb47 2023-04-19 20:54:18,818 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/9bdaa4f67193473b9d899f5802a4eb47, entries=150, sequenceid=37, filesize=11.6 K 2023-04-19 20:54:18,820 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/C/69aa733f9b70435ab3b80dfd86dcdbba as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/69aa733f9b70435ab3b80dfd86dcdbba 2023-04-19 20:54:18,833 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/69aa733f9b70435ab3b80dfd86dcdbba, entries=150, sequenceid=37, filesize=11.6 K 2023-04-19 20:54:18,834 DEBUG [MemStoreFlusher.0] regionserver.StoreScanner(961): StoreScanner already closing. There is no need to updateReaders 2023-04-19 20:54:18,834 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=73.80 KB/75570 for c7ee8087efe9c3ae02af767e46864c85 in 1351ms, sequenceid=37, compaction requested=false 2023-04-19 20:54:18,835 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c7ee8087efe9c3ae02af767e46864c85: 2023-04-19 20:54:18,872 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-19 20:54:18,891 DEBUG [Thread-837] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:54:18,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:54:18,895 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:54:18,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:54:18,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:54:18,896 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:54:18,896 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:18,896 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:54:18,896 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:54:18,897 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:18,897 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:18,898 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:18,898 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:18,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:18,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:18,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:18,899 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:18,899 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:54:18,900 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:18,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:54:18,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:18,902 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:54:18,906 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:54:18,906 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:54:18,906 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:54:18,906 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:18,906 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:54:18,909 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:54:18,909 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:54:18,910 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:18,910 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. started... 2023-04-19 20:54:18,910 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c7ee8087efe9c3ae02af767e46864c85 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-19 20:54:18,911 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c7ee8087efe9c3ae02af767e46864c85, store=A 2023-04-19 20:54:18,911 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:54:18,911 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c7ee8087efe9c3ae02af767e46864c85, store=B 2023-04-19 20:54:18,911 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:54:18,911 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c7ee8087efe9c3ae02af767e46864c85, store=C 2023-04-19 20:54:18,911 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:54:18,958 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230419c8a3ff2f4223499c871ca8284589e75f to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419c8a3ff2f4223499c871ca8284589e75f 2023-04-19 20:54:18,990 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=51, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/A/badae060faaa400ea24261b8d7aedfaf 2023-04-19 20:54:19,014 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=51 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/B/12c4801fea7e4005b2b6d402e1483f57 2023-04-19 20:54:19,040 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=51 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/C/7e79e56e618c487f959113e68f83d37d 2023-04-19 20:54:19,047 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/A/badae060faaa400ea24261b8d7aedfaf as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/badae060faaa400ea24261b8d7aedfaf 2023-04-19 20:54:19,052 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/badae060faaa400ea24261b8d7aedfaf, entries=150, sequenceid=51, filesize=25.1 K 2023-04-19 20:54:19,054 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/B/12c4801fea7e4005b2b6d402e1483f57 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/12c4801fea7e4005b2b6d402e1483f57 2023-04-19 20:54:19,065 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/12c4801fea7e4005b2b6d402e1483f57, entries=150, sequenceid=51, filesize=11.6 K 2023-04-19 20:54:19,065 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/C/7e79e56e618c487f959113e68f83d37d as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/7e79e56e618c487f959113e68f83d37d 2023-04-19 20:54:19,085 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/7e79e56e618c487f959113e68f83d37d, entries=150, sequenceid=51, filesize=11.6 K 2023-04-19 20:54:19,086 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=0 B/0 for c7ee8087efe9c3ae02af767e46864c85 in 175ms, sequenceid=51, compaction requested=true 2023-04-19 20:54:19,086 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c7ee8087efe9c3ae02af767e46864c85: 2023-04-19 20:54:19,086 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:19,086 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:54:19,086 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:54:19,086 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:19,086 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:54:19,086 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:54:19,088 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:19,088 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:19,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:19,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:19,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:19,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:19,089 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:19,089 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:54:19,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:19,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:19,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:19,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:19,092 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:54:19,092 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@987cf23[Count = 0] remaining members to acquire global barrier 2023-04-19 20:54:19,092 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:54:19,092 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:19,094 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:19,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:19,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:19,094 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:54:19,094 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:54:19,094 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:54:19,094 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:19,094 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:54:19,096 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:19,096 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:54:19,096 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:19,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:19,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:19,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:19,096 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:54:19,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:19,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:19,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:19,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:19,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:19,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:19,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:19,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:54:19,099 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:54:19,099 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:54:19,099 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:54:19,099 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:54:19,099 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:54:19,099 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:54:19,101 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:19,101 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:19,101 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:19,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:19,101 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:19,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:19,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:19,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:19,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:19,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:19,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:19,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:19,102 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:19,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:19,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:19,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:19,104 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:19,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:19,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:19,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:19,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:19,107 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:19,108 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:19,108 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:19,108 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:19,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:19,108 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:19,108 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:54:19,108 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:54:19,108 DEBUG [Thread-837] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:54:19,108 DEBUG [Thread-837] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:54:19,108 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:19,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:19,108 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:19,108 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:19,108 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:19,109 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:19,109 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:19,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:19,208 DEBUG [Thread-837] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:54:19,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:54:19,212 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:54:19,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:54:19,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:54:19,213 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:54:19,213 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:19,213 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:54:19,213 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:54:19,214 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:19,214 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:19,215 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:19,215 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:19,215 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:19,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:19,216 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:19,216 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:54:19,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:19,216 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:19,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:54:19,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:19,217 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:54:19,228 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:54:19,228 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:54:19,228 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:54:19,228 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:19,228 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:54:19,228 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:54:19,228 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:54:19,229 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:19,229 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. started... 2023-04-19 20:54:19,229 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c7ee8087efe9c3ae02af767e46864c85: 2023-04-19 20:54:19,229 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:19,229 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:54:19,229 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:54:19,229 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:19,229 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:54:19,229 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:54:19,231 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:19,231 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:19,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:19,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:19,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:19,231 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:19,231 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:54:19,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:19,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:19,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:19,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:19,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:19,233 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:54:19,233 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ee5799b[Count = 0] remaining members to acquire global barrier 2023-04-19 20:54:19,233 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:54:19,233 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:19,234 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:19,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:19,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:19,234 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:54:19,234 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:54:19,234 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:54:19,234 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:19,234 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:54:19,236 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:19,236 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:54:19,236 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:19,236 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:54:19,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:19,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:19,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:19,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:19,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:19,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:19,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:19,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:19,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:19,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:19,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:54:19,240 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:54:19,240 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:54:19,240 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:54:19,240 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:54:19,240 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:54:19,240 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:54:19,242 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:19,242 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:19,242 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:19,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:19,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:19,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:19,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:19,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:19,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:19,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:19,243 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:19,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:19,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:19,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:19,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:19,244 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:19,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:19,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:19,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:19,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:19,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:19,247 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:19,247 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:19,247 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:54:19,247 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:54:19,247 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:19,248 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:19,247 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:19,248 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:19,248 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:19,248 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:19,248 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:19,248 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:19,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:19,248 DEBUG [Thread-837] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:54:19,248 DEBUG [Thread-837] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:54:19,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:19,248 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:54:19,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:19,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:19,348 DEBUG [Thread-837] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:54:19,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:54:19,354 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:54:19,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:54:19,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:54:19,356 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:54:19,356 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:19,356 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:54:19,356 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:54:19,357 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:19,357 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:19,358 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:19,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:19,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:19,359 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:19,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:19,359 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:19,359 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:54:19,359 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:19,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:54:19,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:19,360 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:54:19,360 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:54:19,360 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:54:19,360 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:54:19,360 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:19,360 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:54:19,361 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:54:19,361 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:54:19,361 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:19,361 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. started... 2023-04-19 20:54:19,361 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c7ee8087efe9c3ae02af767e46864c85 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-19 20:54:19,361 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c7ee8087efe9c3ae02af767e46864c85, store=A 2023-04-19 20:54:19,361 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:54:19,361 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c7ee8087efe9c3ae02af767e46864c85, store=B 2023-04-19 20:54:19,361 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:54:19,362 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c7ee8087efe9c3ae02af767e46864c85, store=C 2023-04-19 20:54:19,362 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:54:19,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(9158): Flush requested on c7ee8087efe9c3ae02af767e46864c85 2023-04-19 20:54:19,367 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. as already flushing 2023-04-19 20:54:19,417 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=859, cellSize=39335, totalHeapSize=89576, min timestamp=1681937659362, max timestamp=1681937659415Num uniques 859; 2023-04-19 20:54:19,417 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=859, cellSize=39335, totalHeapSize=89576, min timestamp=1681937659362, max timestamp=1681937659415Num uniques 859; 2023-04-19 20:54:19,417 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=39335, suffix heap size=117064,new segment heap size=89576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=859 2023-04-19 20:54:19,417 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=39335, suffix heap size=117064,new segment heap size=89576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=859 2023-04-19 20:54:19,418 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=859, cellSize=39335, totalHeapSize=89576, min timestamp=1681937659362, max timestamp=1681937659415Num uniques 859; 2023-04-19 20:54:19,418 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=39335, suffix heap size=117064,new segment heap size=89576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=859 2023-04-19 20:54:19,440 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:19,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56352 deadline: 1681937719436, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:19,441 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:19,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56384 deadline: 1681937719437, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:19,542 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:19,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56352 deadline: 1681937719542, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:19,542 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:19,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56384 deadline: 1681937719542, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:19,617 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:19,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56326 deadline: 1681937719616, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:19,618 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:19,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56408 deadline: 1681937719617, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:19,622 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:19,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56368 deadline: 1681937719619, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:19,746 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:19,746 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:19,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56384 deadline: 1681937719743, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:19,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56352 deadline: 1681937719744, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:19,786 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304199166b8188ce442d49557ff1369e59dbb to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304199166b8188ce442d49557ff1369e59dbb 2023-04-19 20:54:20,047 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:20,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56384 deadline: 1681937720047, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:20,048 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:20,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56352 deadline: 1681937720047, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:20,192 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=60, memsize=11.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/A/eb9caf8a5ede47a4a488c2d07dc15176 2023-04-19 20:54:20,208 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=60 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/B/c50116b3f4d64ea8bb8aeb45caeef088 2023-04-19 20:54:20,225 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=60 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/C/639d44ae22504a589807fcadc1e8e230 2023-04-19 20:54:20,230 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/A/eb9caf8a5ede47a4a488c2d07dc15176 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/eb9caf8a5ede47a4a488c2d07dc15176 2023-04-19 20:54:20,234 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/eb9caf8a5ede47a4a488c2d07dc15176, entries=250, sequenceid=60, filesize=38.7 K 2023-04-19 20:54:20,236 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/B/c50116b3f4d64ea8bb8aeb45caeef088 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/c50116b3f4d64ea8bb8aeb45caeef088 2023-04-19 20:54:20,242 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/c50116b3f4d64ea8bb8aeb45caeef088, entries=150, sequenceid=60, filesize=11.6 K 2023-04-19 20:54:20,245 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/C/639d44ae22504a589807fcadc1e8e230 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/639d44ae22504a589807fcadc1e8e230 2023-04-19 20:54:20,250 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/639d44ae22504a589807fcadc1e8e230, entries=150, sequenceid=60, filesize=11.6 K 2023-04-19 20:54:20,251 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=167.72 KB/171750 for c7ee8087efe9c3ae02af767e46864c85 in 890ms, sequenceid=60, compaction requested=true 2023-04-19 20:54:20,251 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c7ee8087efe9c3ae02af767e46864c85: 2023-04-19 20:54:20,251 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:20,252 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:54:20,252 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:54:20,252 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:20,252 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:54:20,252 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:54:20,254 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:20,254 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:20,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:20,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:20,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:20,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:20,255 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:20,255 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:54:20,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:20,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:20,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:20,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:20,257 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:54:20,258 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2f5b598e[Count = 0] remaining members to acquire global barrier 2023-04-19 20:54:20,258 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:54:20,258 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:20,259 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:20,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:20,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:20,259 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:54:20,259 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:54:20,259 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:54:20,259 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:20,259 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:54:20,261 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:54:20,261 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:20,261 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:54:20,262 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:20,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:20,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:20,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:20,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:20,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:20,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:20,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:20,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:20,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:20,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:20,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:54:20,268 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:54:20,268 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:54:20,268 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:54:20,268 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:54:20,268 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:54:20,268 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:54:20,270 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:20,270 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:20,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:20,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:20,271 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:20,271 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:20,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:20,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:20,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:20,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:20,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:20,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:20,272 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:20,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:20,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:20,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:20,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:20,273 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:20,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:20,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:20,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:20,278 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:20,278 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:20,278 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:20,278 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:20,278 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:20,278 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:20,278 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:20,278 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:20,278 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:20,278 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:20,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:20,278 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:20,278 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:54:20,278 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:54:20,278 DEBUG [Thread-837] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:54:20,278 DEBUG [Thread-837] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:54:20,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:20,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:20,379 DEBUG [Thread-837] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:54:20,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:54:20,381 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:54:20,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:54:20,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:54:20,382 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:54:20,382 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:20,382 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:54:20,382 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:54:20,383 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:20,383 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:20,385 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:20,385 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:20,385 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:20,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:20,385 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:20,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:20,385 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:54:20,385 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:20,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:54:20,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:20,386 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:54:20,386 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:54:20,386 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:54:20,386 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:54:20,387 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:20,387 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:54:20,387 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:54:20,388 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:54:20,388 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:20,388 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. started... 2023-04-19 20:54:20,388 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c7ee8087efe9c3ae02af767e46864c85 3/3 column families, dataSize=167.72 KB heapSize=418.97 KB 2023-04-19 20:54:20,388 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c7ee8087efe9c3ae02af767e46864c85, store=A 2023-04-19 20:54:20,388 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 20:54:20,388 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c7ee8087efe9c3ae02af767e46864c85, store=B 2023-04-19 20:54:20,388 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 20:54:20,388 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c7ee8087efe9c3ae02af767e46864c85, store=C 2023-04-19 20:54:20,388 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 20:54:20,420 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304198cc55b9bc9af4fa1b34b9d25f023aee2 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304198cc55b9bc9af4fa1b34b9d25f023aee2 2023-04-19 20:54:20,426 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=88, memsize=55.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/A/57bec31be5684a0f92cb8f2d7c0262fb 2023-04-19 20:54:20,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(9158): Flush requested on c7ee8087efe9c3ae02af767e46864c85 2023-04-19 20:54:20,554 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. as already flushing 2023-04-19 20:54:20,562 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:20,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56352 deadline: 1681937720561, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:20,564 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:20,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56384 deadline: 1681937720562, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:20,665 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:20,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56352 deadline: 1681937720663, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:20,666 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:20,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56384 deadline: 1681937720665, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:20,849 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=55.91 KB at sequenceid=88 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/B/009532cfbe874673a4492025c0b126cc 2023-04-19 20:54:20,867 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:20,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56352 deadline: 1681937720866, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:20,867 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:20,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56384 deadline: 1681937720867, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:21,169 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:21,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56352 deadline: 1681937721168, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:21,172 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:21,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56384 deadline: 1681937721169, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:21,262 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=55.91 KB at sequenceid=88 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/C/e01e4a7c480242428ae5667c87c4b5d7 2023-04-19 20:54:21,267 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/A/57bec31be5684a0f92cb8f2d7c0262fb as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/57bec31be5684a0f92cb8f2d7c0262fb 2023-04-19 20:54:21,272 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/57bec31be5684a0f92cb8f2d7c0262fb, entries=150, sequenceid=88, filesize=25.1 K 2023-04-19 20:54:21,273 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/B/009532cfbe874673a4492025c0b126cc as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/009532cfbe874673a4492025c0b126cc 2023-04-19 20:54:21,284 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/009532cfbe874673a4492025c0b126cc, entries=150, sequenceid=88, filesize=11.6 K 2023-04-19 20:54:21,287 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/C/e01e4a7c480242428ae5667c87c4b5d7 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/e01e4a7c480242428ae5667c87c4b5d7 2023-04-19 20:54:21,291 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/e01e4a7c480242428ae5667c87c4b5d7, entries=150, sequenceid=88, filesize=11.6 K 2023-04-19 20:54:21,292 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~167.72 KB/171750, heapSize ~418.92 KB/428976, currentSize=40.25 KB/41220 for c7ee8087efe9c3ae02af767e46864c85 in 904ms, sequenceid=88, compaction requested=true 2023-04-19 20:54:21,292 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c7ee8087efe9c3ae02af767e46864c85: 2023-04-19 20:54:21,292 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:21,293 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:54:21,293 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:54:21,293 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:21,293 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:54:21,293 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:54:21,295 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:21,295 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:21,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:21,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:21,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:21,295 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:21,295 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:54:21,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:21,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:21,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:21,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:21,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:21,296 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:54:21,297 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@79d08aaf[Count = 0] remaining members to acquire global barrier 2023-04-19 20:54:21,297 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:54:21,297 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:21,298 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:21,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:21,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:21,298 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:54:21,298 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:54:21,298 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:54:21,298 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:21,298 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:54:21,300 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:21,300 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:54:21,300 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:21,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:21,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:21,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:21,300 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:54:21,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:21,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:21,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:21,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:21,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:21,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:21,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:21,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:54:21,303 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:54:21,303 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:54:21,303 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:54:21,303 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:54:21,303 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:54:21,303 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:54:21,305 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:21,305 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:21,305 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:21,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:21,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:21,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:21,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:21,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:21,306 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:21,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:21,306 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:21,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:21,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:21,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:21,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:21,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:21,307 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:21,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:21,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:21,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:21,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:21,312 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:21,313 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:21,313 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:21,313 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:21,313 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:21,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:21,313 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:54:21,313 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:54:21,313 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:21,313 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:21,313 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:21,313 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:21,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:21,313 DEBUG [Thread-837] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:54:21,313 DEBUG [Thread-837] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:54:21,313 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:21,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:21,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:21,413 DEBUG [Thread-837] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:54:21,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:54:21,416 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:54:21,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:54:21,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:54:21,420 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:54:21,420 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:21,420 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:54:21,420 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:54:21,421 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:21,421 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:21,423 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:21,423 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:21,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:21,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:21,423 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:21,423 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:54:21,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:21,424 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:21,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:54:21,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:21,424 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:54:21,428 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:54:21,428 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:54:21,428 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:54:21,428 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:21,428 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:54:21,428 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:54:21,428 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:54:21,428 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:21,429 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. started... 2023-04-19 20:54:21,429 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c7ee8087efe9c3ae02af767e46864c85 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-19 20:54:21,429 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c7ee8087efe9c3ae02af767e46864c85, store=A 2023-04-19 20:54:21,429 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:54:21,429 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c7ee8087efe9c3ae02af767e46864c85, store=B 2023-04-19 20:54:21,429 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:54:21,429 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c7ee8087efe9c3ae02af767e46864c85, store=C 2023-04-19 20:54:21,429 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:54:21,459 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230419721296777b43468ab8a16455e46e1cf5 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419721296777b43468ab8a16455e46e1cf5 2023-04-19 20:54:21,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(9158): Flush requested on c7ee8087efe9c3ae02af767e46864c85 2023-04-19 20:54:21,630 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. as already flushing 2023-04-19 20:54:21,666 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681937661656, max timestamp=1681937661664Num uniques 859; 2023-04-19 20:54:21,666 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-19 20:54:21,666 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681937661656, max timestamp=1681937661664Num uniques 859; 2023-04-19 20:54:21,666 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-19 20:54:21,666 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681937661656, max timestamp=1681937661664Num uniques 859; 2023-04-19 20:54:21,666 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-19 20:54:21,704 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:21,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56326 deadline: 1681937721697, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:21,705 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:21,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56368 deadline: 1681937721697, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:21,705 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:21,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56408 deadline: 1681937721697, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:21,705 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:21,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56384 deadline: 1681937721698, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:21,705 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:21,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56352 deadline: 1681937721698, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:21,807 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:21,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56326 deadline: 1681937721806, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:21,807 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:21,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56368 deadline: 1681937721806, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:21,807 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:21,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56408 deadline: 1681937721806, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:21,808 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:21,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56384 deadline: 1681937721806, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:21,809 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:21,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56352 deadline: 1681937721806, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:21,865 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=97, memsize=13.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/A/31061a94ffe443c1973cedefddf32e35 2023-04-19 20:54:22,011 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:22,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56326 deadline: 1681937722008, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:22,011 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:22,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56408 deadline: 1681937722008, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:22,012 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:22,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56368 deadline: 1681937722009, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:22,014 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:22,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56384 deadline: 1681937722012, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:22,014 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:22,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56352 deadline: 1681937722012, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:22,277 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=97 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/B/6df4a594e7594ceb83c1946f21123132 2023-04-19 20:54:22,314 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:22,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56368 deadline: 1681937722313, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:22,314 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:22,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56326 deadline: 1681937722313, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:22,316 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:22,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56352 deadline: 1681937722315, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:22,319 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:22,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56384 deadline: 1681937722317, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:22,325 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:22,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56408 deadline: 1681937722324, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:22,691 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=97 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/C/b963dd73018b49d78c8e3816f6e38fd1 2023-04-19 20:54:22,696 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/A/31061a94ffe443c1973cedefddf32e35 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/31061a94ffe443c1973cedefddf32e35 2023-04-19 20:54:22,700 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/31061a94ffe443c1973cedefddf32e35, entries=150, sequenceid=97, filesize=25.1 K 2023-04-19 20:54:22,701 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/B/6df4a594e7594ceb83c1946f21123132 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/6df4a594e7594ceb83c1946f21123132 2023-04-19 20:54:22,711 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/6df4a594e7594ceb83c1946f21123132, entries=150, sequenceid=97, filesize=11.6 K 2023-04-19 20:54:22,712 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/C/b963dd73018b49d78c8e3816f6e38fd1 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/b963dd73018b49d78c8e3816f6e38fd1 2023-04-19 20:54:22,717 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/b963dd73018b49d78c8e3816f6e38fd1, entries=150, sequenceid=97, filesize=11.6 K 2023-04-19 20:54:22,717 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=140.89 KB/144270 for c7ee8087efe9c3ae02af767e46864c85 in 1288ms, sequenceid=97, compaction requested=true 2023-04-19 20:54:22,718 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c7ee8087efe9c3ae02af767e46864c85: 2023-04-19 20:54:22,718 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:22,718 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:54:22,718 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:54:22,718 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:22,718 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:54:22,718 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:54:22,719 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:22,719 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:22,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:22,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:22,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:22,720 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:22,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:22,720 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:54:22,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:22,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:22,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:22,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:22,721 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:54:22,722 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4e19bc61[Count = 0] remaining members to acquire global barrier 2023-04-19 20:54:22,722 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:54:22,722 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:22,723 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:22,723 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:22,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:22,723 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:54:22,723 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:54:22,723 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:54:22,723 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:22,723 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:54:22,724 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:54:22,724 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:22,724 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:22,724 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:54:22,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:22,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:22,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:22,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:22,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:22,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:22,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:22,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:22,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:22,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:22,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:54:22,727 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:54:22,727 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:54:22,727 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:54:22,727 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:54:22,727 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:54:22,727 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:54:22,729 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:22,729 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:22,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:22,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:22,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:22,730 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:22,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:22,730 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:22,730 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:22,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:22,730 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:22,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:22,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:22,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:22,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:22,731 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:22,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:22,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:22,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:22,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:22,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:22,734 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:22,734 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:22,734 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:22,734 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:22,734 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:22,734 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:22,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:22,734 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:22,734 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:22,734 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:22,734 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:22,734 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:22,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:22,735 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:54:22,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:22,744 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:54:22,744 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:54:22,744 DEBUG [Thread-837] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:54:22,744 DEBUG [Thread-837] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:54:22,822 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681937661692, max timestamp=1681937662819Num uniques 1068; 2023-04-19 20:54:22,822 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681937661692, max timestamp=1681937662819Num uniques 1068; 2023-04-19 20:54:22,822 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-04-19 20:54:22,822 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681937661692, max timestamp=1681937662820Num uniques 1068; 2023-04-19 20:54:22,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(9158): Flush requested on c7ee8087efe9c3ae02af767e46864c85 2023-04-19 20:54:22,823 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-04-19 20:54:22,823 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c7ee8087efe9c3ae02af767e46864c85 3/3 column families, dataSize=33.54 KB heapSize=85.31 KB 2023-04-19 20:54:22,823 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-04-19 20:54:22,824 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c7ee8087efe9c3ae02af767e46864c85, store=A 2023-04-19 20:54:22,824 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 20:54:22,824 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c7ee8087efe9c3ae02af767e46864c85, store=B 2023-04-19 20:54:22,824 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 20:54:22,824 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c7ee8087efe9c3ae02af767e46864c85, store=C 2023-04-19 20:54:22,824 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 20:54:22,844 DEBUG [Thread-837] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:54:22,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:54:22,846 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:54:22,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:54:22,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:54:22,848 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:54:22,848 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:22,848 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:54:22,848 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:54:22,848 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:22,848 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:22,850 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:22,850 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:22,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:22,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:22,850 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:22,850 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:54:22,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:22,850 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:22,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:54:22,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:22,851 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:54:22,851 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:54:22,851 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:54:22,851 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:54:22,851 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:22,851 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:54:22,852 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:54:22,852 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:54:22,852 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:22,852 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. started... 2023-04-19 20:54:22,852 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. as already flushing 2023-04-19 20:54:22,852 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:22,852 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:54:22,852 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:54:22,852 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:22,852 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:54:22,852 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1681937662840, max timestamp=1681937662850Num uniques 859; 2023-04-19 20:54:22,852 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-04-19 20:54:22,852 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:54:22,852 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1681937662840, max timestamp=1681937662850Num uniques 859; 2023-04-19 20:54:22,853 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-04-19 20:54:22,853 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1681937662840, max timestamp=1681937662850Num uniques 859; 2023-04-19 20:54:22,853 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-04-19 20:54:22,854 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:22,854 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:22,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:22,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:22,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:22,855 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:22,855 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:54:22,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:22,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:22,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:22,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:22,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:22,856 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:54:22,856 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1be52d98[Count = 0] remaining members to acquire global barrier 2023-04-19 20:54:22,856 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:54:22,857 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:22,859 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:22,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:22,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:22,859 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:54:22,859 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:54:22,859 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:54:22,859 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:22,859 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:54:22,861 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:22,861 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:54:22,861 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:22,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:22,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:22,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:22,861 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:54:22,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:22,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:22,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:22,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:22,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:22,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:22,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:22,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:54:22,863 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:54:22,864 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:54:22,864 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:54:22,864 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:54:22,864 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:54:22,864 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:54:22,865 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:22,865 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:22,865 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:22,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:22,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:22,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:22,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:22,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:22,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:22,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:22,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:22,867 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:22,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:22,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:22,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:22,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:22,869 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:22,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:22,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:22,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:22,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:22,871 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:22,871 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:22,871 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:22,871 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:54:22,871 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:54:22,871 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:22,871 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:22,871 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:22,871 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:22,871 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:54:22,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:22,871 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:22,871 DEBUG [Thread-837] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:54:22,871 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:22,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:22,871 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:22,871 DEBUG [Thread-837] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:54:22,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:22,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:22,877 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:22,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56326 deadline: 1681937722869, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:22,881 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:22,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56408 deadline: 1681937722874, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:22,881 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:22,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56384 deadline: 1681937722875, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:22,882 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:22,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56352 deadline: 1681937722876, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:22,882 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:22,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56368 deadline: 1681937722877, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:22,972 DEBUG [Thread-837] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:54:22,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:54:22,974 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:54:22,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:54:22,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:54:22,975 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:54:22,975 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:22,975 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:54:22,975 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:54:22,976 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:22,976 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:22,978 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:22,978 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:22,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:22,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:22,978 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:22,978 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:54:22,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:22,979 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:22,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:54:22,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:22,979 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:54:22,980 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:54:22,980 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:54:22,980 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:54:22,980 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:22,980 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:54:22,980 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:54:22,980 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:54:22,980 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:22,980 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. started... 2023-04-19 20:54:22,980 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. as already flushing 2023-04-19 20:54:22,980 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:22,980 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:54:22,980 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:54:22,980 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:22,980 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:54:22,980 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:54:22,981 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:22,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56326 deadline: 1681937722978, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:22,982 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:22,982 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:22,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:22,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:22,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:22,983 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:22,983 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:54:22,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:22,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:22,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:22,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:22,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:22,984 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:54:22,984 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6e879043[Count = 0] remaining members to acquire global barrier 2023-04-19 20:54:22,984 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:54:22,984 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:22,985 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:22,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56408 deadline: 1681937722982, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:22,985 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:22,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56384 deadline: 1681937722982, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:22,985 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:22,986 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:22,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:22,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:22,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56352 deadline: 1681937722983, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:22,986 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:22,986 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:54:22,986 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:54:22,986 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:54:22,986 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:22,986 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:54:22,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56368 deadline: 1681937722983, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:22,987 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:22,987 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:54:22,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:22,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:22,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:22,987 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:22,988 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:54:22,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:22,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:22,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:22,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:22,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:22,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:22,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:22,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:54:22,990 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:54:22,990 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:54:22,990 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:54:22,990 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:54:22,990 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:54:22,990 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:54:22,996 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:22,996 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:22,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:22,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:22,996 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:22,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:22,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:22,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:22,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:22,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:22,997 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:22,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:22,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:22,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:22,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:22,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:22,998 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:22,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:22,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:22,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:22,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:23,000 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:23,000 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:23,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:23,000 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:54:23,000 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:54:23,000 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:23,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:23,000 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:54:23,000 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:23,000 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:23,000 DEBUG [Thread-837] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:54:23,000 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:23,000 DEBUG [Thread-837] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:54:23,000 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:23,000 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:23,000 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:23,000 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:23,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:23,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:23,100 DEBUG [Thread-837] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:54:23,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:54:23,103 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:54:23,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:54:23,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:54:23,104 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:54:23,104 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:23,104 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:54:23,104 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:54:23,105 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:23,105 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:23,110 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:23,110 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:23,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:23,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:23,110 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:23,110 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:54:23,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:23,110 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:23,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:54:23,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:23,111 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:54:23,111 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:54:23,111 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:54:23,111 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:54:23,111 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:23,111 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:54:23,111 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:54:23,112 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:54:23,112 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:23,112 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. started... 2023-04-19 20:54:23,112 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. as already flushing 2023-04-19 20:54:23,112 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:23,112 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:54:23,112 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:54:23,112 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:23,112 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:54:23,112 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:54:23,117 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:23,117 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:23,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:23,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:23,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:23,117 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:23,117 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:54:23,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:23,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:23,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:23,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:23,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:23,119 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:54:23,119 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:54:23,119 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@137fface[Count = 0] remaining members to acquire global barrier 2023-04-19 20:54:23,119 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:23,120 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:23,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:23,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:23,120 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:54:23,120 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:54:23,120 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:54:23,121 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:23,121 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:54:23,122 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:23,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:23,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:23,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:23,122 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:54:23,122 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:23,122 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:54:23,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:23,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:23,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:23,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:23,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:23,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:23,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:23,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:54:23,124 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:54:23,124 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:54:23,124 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:54:23,124 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:54:23,124 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:54:23,124 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:54:23,126 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:23,126 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:23,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:23,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:23,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:23,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:23,126 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:23,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:23,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:23,127 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:23,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:23,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:23,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:23,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:23,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:23,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:23,128 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:23,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:23,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:23,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:23,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:23,132 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:23,132 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:23,132 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:23,132 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:54:23,132 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:54:23,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:23,132 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:23,132 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:23,132 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:23,133 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:23,133 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:23,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:23,133 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:23,133 DEBUG [Thread-837] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:54:23,133 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:23,133 DEBUG [Thread-837] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:54:23,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:23,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:23,184 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:23,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56326 deadline: 1681937723182, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:23,189 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:23,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56384 deadline: 1681937723186, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:23,189 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:23,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56368 deadline: 1681937723187, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:23,189 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:23,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56408 deadline: 1681937723187, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:23,189 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:23,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56352 deadline: 1681937723188, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:23,233 DEBUG [Thread-837] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:54:23,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:54:23,235 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:54:23,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:54:23,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:54:23,237 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:54:23,237 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:23,237 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:54:23,237 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:54:23,237 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:23,237 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:23,239 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:23,239 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:23,239 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:23,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:23,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:23,239 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:23,240 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:54:23,240 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:23,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:54:23,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:23,240 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:54:23,241 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:54:23,241 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:54:23,241 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:54:23,241 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:23,241 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:54:23,241 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:54:23,241 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:54:23,241 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:23,241 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. started... 2023-04-19 20:54:23,241 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. as already flushing 2023-04-19 20:54:23,241 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:23,241 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:54:23,242 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:54:23,242 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:23,242 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:54:23,242 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:54:23,243 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:23,243 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304191874794a078f4516921021818046cd2c to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304191874794a078f4516921021818046cd2c 2023-04-19 20:54:23,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:23,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:23,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:23,244 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:23,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:23,244 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:23,244 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:54:23,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:23,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:23,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:23,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:23,246 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:54:23,246 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1086a674[Count = 0] remaining members to acquire global barrier 2023-04-19 20:54:23,247 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:54:23,247 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:23,248 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:23,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:23,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:23,248 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:54:23,249 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:54:23,249 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:23,249 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:54:23,249 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:54:23,251 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:54:23,251 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:23,251 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:23,251 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:54:23,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:23,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:23,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:23,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:23,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:23,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:23,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:23,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:23,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:23,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:23,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:54:23,254 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:54:23,254 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:54:23,254 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:54:23,254 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:54:23,254 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:54:23,254 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:54:23,256 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:23,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:23,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:23,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:23,256 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:23,256 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:23,256 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:23,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:23,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:23,256 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:23,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:23,257 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:23,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:23,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:23,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:23,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:23,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:23,259 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:23,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:23,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:23,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:23,261 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:23,261 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:23,261 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:23,261 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:54:23,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:23,261 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:54:23,261 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:54:23,261 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:23,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:23,261 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:23,261 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:23,261 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:23,261 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:23,261 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:23,261 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:23,261 DEBUG [Thread-837] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:54:23,262 DEBUG [Thread-837] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:54:23,262 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:23,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:23,362 DEBUG [Thread-837] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:54:23,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:54:23,364 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:54:23,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:54:23,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:54:23,367 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:54:23,367 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:23,367 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:54:23,367 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:54:23,367 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:23,367 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:23,368 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:23,369 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:23,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:23,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:23,369 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:23,369 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:54:23,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:23,369 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:23,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:54:23,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:23,370 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:54:23,371 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:54:23,371 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:54:23,371 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:54:23,371 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:23,371 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:54:23,371 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:54:23,371 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:54:23,371 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:23,371 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. started... 2023-04-19 20:54:23,371 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. as already flushing 2023-04-19 20:54:23,371 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:23,372 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:54:23,372 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:54:23,372 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:23,372 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:54:23,372 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:54:23,374 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:23,374 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:23,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:23,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:23,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:23,375 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:23,375 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:54:23,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:23,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:23,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:23,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:23,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:23,377 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:54:23,377 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@223dd011[Count = 0] remaining members to acquire global barrier 2023-04-19 20:54:23,377 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:54:23,377 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:23,378 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:23,378 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:23,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:23,378 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:54:23,378 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:54:23,378 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:54:23,378 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:23,378 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:54:23,380 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:23,380 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:54:23,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:23,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:23,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:23,380 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:23,380 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:54:23,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:23,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:23,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:23,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:23,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:23,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:23,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:23,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:54:23,383 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:54:23,383 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:54:23,383 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:54:23,383 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:54:23,383 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:54:23,383 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:54:23,385 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:23,385 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:23,385 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:23,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:23,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:23,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:23,385 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:23,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:23,386 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:23,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:23,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:23,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:23,387 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:23,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:23,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:23,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:23,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:23,388 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:23,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:23,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:23,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:23,396 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:23,396 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:23,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:23,396 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:23,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:23,396 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:54:23,396 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:54:23,396 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:23,396 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:23,396 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:23,396 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:23,396 DEBUG [Thread-837] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:54:23,396 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:23,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:23,396 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:23,396 DEBUG [Thread-837] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:54:23,396 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:23,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:23,488 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:23,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56326 deadline: 1681937723485, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:23,492 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:23,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56368 deadline: 1681937723490, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:23,493 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:23,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56352 deadline: 1681937723490, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:23,494 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:23,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56408 deadline: 1681937723491, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:23,494 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:23,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56384 deadline: 1681937723492, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:23,496 DEBUG [Thread-837] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:54:23,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:54:23,499 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:54:23,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:54:23,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:54:23,500 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:54:23,500 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:23,500 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:54:23,500 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:54:23,500 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:23,500 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:23,502 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:23,502 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:23,502 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:23,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:23,502 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:23,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:23,502 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:54:23,503 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:23,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:54:23,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:23,503 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:54:23,503 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:54:23,503 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:54:23,503 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:54:23,503 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:23,503 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:54:23,504 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:54:23,504 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:54:23,504 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:23,504 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. started... 2023-04-19 20:54:23,504 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. as already flushing 2023-04-19 20:54:23,504 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:23,504 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:54:23,504 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:54:23,504 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:23,504 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:54:23,504 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:54:23,506 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:23,506 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:23,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:23,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:23,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:23,506 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:23,506 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:54:23,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:23,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:23,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:23,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:23,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:23,507 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:54:23,507 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@70ea499d[Count = 0] remaining members to acquire global barrier 2023-04-19 20:54:23,507 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:54:23,508 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:23,509 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:23,509 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:23,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:23,509 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:54:23,509 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:54:23,509 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:54:23,509 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:23,509 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:54:23,513 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:23,513 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:54:23,513 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:23,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:23,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:23,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:23,513 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:54:23,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:23,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:23,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:23,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:23,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:23,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:23,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:23,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:54:23,516 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:54:23,516 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:54:23,516 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:54:23,516 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:54:23,516 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:54:23,516 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:54:23,518 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:23,518 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:23,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:23,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:23,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:23,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:23,518 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:23,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:23,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:23,518 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:23,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:23,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:23,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:23,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:23,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:23,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:23,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:23,520 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:23,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:23,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:23,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:23,522 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:23,522 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:23,522 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:23,522 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:23,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:23,522 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:23,522 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:23,522 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:54:23,522 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:54:23,522 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:23,522 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:23,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:23,522 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:23,523 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:23,523 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:23,523 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:23,523 DEBUG [Thread-837] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:54:23,523 DEBUG [Thread-837] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:54:23,623 DEBUG [Thread-837] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:54:23,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:54:23,625 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:54:23,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:54:23,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:54:23,626 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:54:23,626 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:23,627 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:54:23,627 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:54:23,627 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:23,627 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:23,628 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:23,628 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:23,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:23,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:23,629 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:23,629 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:54:23,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:23,629 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:23,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:54:23,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:23,629 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:54:23,630 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:54:23,630 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:54:23,630 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:54:23,630 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:23,630 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:54:23,630 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:54:23,630 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:54:23,630 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:23,630 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. started... 2023-04-19 20:54:23,630 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. as already flushing 2023-04-19 20:54:23,630 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:23,630 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:54:23,630 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:54:23,630 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:23,630 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:54:23,630 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:54:23,632 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:23,632 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:23,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:23,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:23,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:23,633 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:23,633 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:54:23,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:23,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:23,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:23,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:23,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:23,636 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:54:23,636 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:54:23,636 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1c2f64c4[Count = 0] remaining members to acquire global barrier 2023-04-19 20:54:23,636 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:23,637 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:23,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:23,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:23,637 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:54:23,637 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:54:23,637 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:54:23,637 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:23,637 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:54:23,639 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:54:23,639 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:23,639 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:23,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:23,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:23,639 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:54:23,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:23,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:23,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:23,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:23,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:23,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:23,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:23,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:23,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:54:23,642 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:54:23,642 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:54:23,642 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:54:23,642 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:54:23,642 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:54:23,642 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:54:23,643 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:23,643 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:23,643 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:23,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:23,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:23,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:23,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:23,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:23,644 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:23,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:23,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:23,644 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:23,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:23,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:23,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:23,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:23,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:23,645 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:23,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:23,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:23,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:23,649 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:23,649 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:23,649 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:23,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:23,649 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:23,649 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:54:23,649 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:54:23,649 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:54:23,649 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:23,649 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:23,649 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:23,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:23,649 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:23,649 DEBUG [Thread-837] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:54:23,649 DEBUG [Thread-837] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:54:23,649 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:23,650 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:23,650 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:23,650 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:23,651 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=137, memsize=13.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/A/3c3e2c86178e4d6e9748dcb76dc206ea 2023-04-19 20:54:23,664 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=137 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/B/35655288ccca4a5d9569eaef5c019ea4 2023-04-19 20:54:23,675 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=137 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/C/6706bc3aa4054b93b3a4d6dec1537c3f 2023-04-19 20:54:23,679 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/A/3c3e2c86178e4d6e9748dcb76dc206ea as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/3c3e2c86178e4d6e9748dcb76dc206ea 2023-04-19 20:54:23,683 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/3c3e2c86178e4d6e9748dcb76dc206ea, entries=300, sequenceid=137, filesize=45.8 K 2023-04-19 20:54:23,684 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/B/35655288ccca4a5d9569eaef5c019ea4 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/35655288ccca4a5d9569eaef5c019ea4 2023-04-19 20:54:23,689 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/35655288ccca4a5d9569eaef5c019ea4, entries=150, sequenceid=137, filesize=11.8 K 2023-04-19 20:54:23,690 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/C/6706bc3aa4054b93b3a4d6dec1537c3f as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/6706bc3aa4054b93b3a4d6dec1537c3f 2023-04-19 20:54:23,695 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/6706bc3aa4054b93b3a4d6dec1537c3f, entries=150, sequenceid=137, filesize=11.8 K 2023-04-19 20:54:23,695 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~105.19 KB/107712, currentSize=154.31 KB/158010 for c7ee8087efe9c3ae02af767e46864c85 in 872ms, sequenceid=137, compaction requested=true 2023-04-19 20:54:23,695 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c7ee8087efe9c3ae02af767e46864c85: 2023-04-19 20:54:23,696 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 20:54:23,696 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 20:54:23,696 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-19 20:54:23,696 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 20:54:23,696 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-19 20:54:23,702 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 83352 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-19 20:54:23,702 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 222170 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-19 20:54:23,702 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1912): c7ee8087efe9c3ae02af767e46864c85/B is initiating minor compaction (all files) 2023-04-19 20:54:23,702 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1912): c7ee8087efe9c3ae02af767e46864c85/A is initiating minor compaction (all files) 2023-04-19 20:54:23,702 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c7ee8087efe9c3ae02af767e46864c85/B in TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:23,702 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c7ee8087efe9c3ae02af767e46864c85/A in TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:23,702 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/03bec7be52ca4b7c9d6044c713757b35, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/101e02bda7a447a5ba7839a7ea2044cf, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/badae060faaa400ea24261b8d7aedfaf, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/eb9caf8a5ede47a4a488c2d07dc15176, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/57bec31be5684a0f92cb8f2d7c0262fb, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/31061a94ffe443c1973cedefddf32e35, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/3c3e2c86178e4d6e9748dcb76dc206ea] into tmpdir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp, totalSize=217.0 K 2023-04-19 20:54:23,703 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/0aba3f28101349eb99fb5dbb80cdfd55, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/9bdaa4f67193473b9d899f5802a4eb47, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/12c4801fea7e4005b2b6d402e1483f57, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/c50116b3f4d64ea8bb8aeb45caeef088, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/009532cfbe874673a4492025c0b126cc, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/6df4a594e7594ceb83c1946f21123132, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/35655288ccca4a5d9569eaef5c019ea4] into tmpdir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp, totalSize=81.4 K 2023-04-19 20:54:23,703 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 03bec7be52ca4b7c9d6044c713757b35, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=18, earliestPutTs=1681937657135 2023-04-19 20:54:23,703 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 0aba3f28101349eb99fb5dbb80cdfd55, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=18, earliestPutTs=1681937657135 2023-04-19 20:54:23,703 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 101e02bda7a447a5ba7839a7ea2044cf, keycount=200, bloomtype=ROW, size=31.9 K, encoding=NONE, compression=NONE, seqNum=37, earliestPutTs=1681937657162 2023-04-19 20:54:23,704 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 9bdaa4f67193473b9d899f5802a4eb47, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=37, earliestPutTs=1681937657162 2023-04-19 20:54:23,704 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting badae060faaa400ea24261b8d7aedfaf, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=51, earliestPutTs=1681937657495 2023-04-19 20:54:23,705 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 12c4801fea7e4005b2b6d402e1483f57, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=51, earliestPutTs=1681937657495 2023-04-19 20:54:23,705 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting eb9caf8a5ede47a4a488c2d07dc15176, keycount=250, bloomtype=ROW, size=38.7 K, encoding=NONE, compression=NONE, seqNum=60, earliestPutTs=1681937659346 2023-04-19 20:54:23,705 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting c50116b3f4d64ea8bb8aeb45caeef088, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=60, earliestPutTs=1681937659353 2023-04-19 20:54:23,706 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 57bec31be5684a0f92cb8f2d7c0262fb, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=88, earliestPutTs=1681937659429 2023-04-19 20:54:23,706 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 009532cfbe874673a4492025c0b126cc, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=88, earliestPutTs=1681937659429 2023-04-19 20:54:23,706 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 31061a94ffe443c1973cedefddf32e35, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=97, earliestPutTs=1681937660552 2023-04-19 20:54:23,706 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 6df4a594e7594ceb83c1946f21123132, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=97, earliestPutTs=1681937660552 2023-04-19 20:54:23,706 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 3c3e2c86178e4d6e9748dcb76dc206ea, keycount=300, bloomtype=ROW, size=45.8 K, encoding=NONE, compression=NONE, seqNum=137, earliestPutTs=1681937661692 2023-04-19 20:54:23,707 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 35655288ccca4a5d9569eaef5c019ea4, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=137, earliestPutTs=1681937661696 2023-04-19 20:54:23,734 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] throttle.PressureAwareThroughputController(145): c7ee8087efe9c3ae02af767e46864c85#B#compaction#127 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-19 20:54:23,742 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] throttle.PressureAwareThroughputController(145): c7ee8087efe9c3ae02af767e46864c85#A#compaction#126 average throughput is 1.10 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 20:54:23,750 DEBUG [Thread-837] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:54:23,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:54:23,754 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:54:23,758 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/B/e944bb05f5534a5fb5d3caa0bdea0e1e as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/e944bb05f5534a5fb5d3caa0bdea0e1e 2023-04-19 20:54:23,764 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in c7ee8087efe9c3ae02af767e46864c85/B of c7ee8087efe9c3ae02af767e46864c85 into e944bb05f5534a5fb5d3caa0bdea0e1e(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 20:54:23,764 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c7ee8087efe9c3ae02af767e46864c85: 2023-04-19 20:54:23,764 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85., storeName=c7ee8087efe9c3ae02af767e46864c85/B, priority=9, startTime=1681937663696; duration=0sec 2023-04-19 20:54:23,764 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 20:54:23,764 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-19 20:54:23,766 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 83352 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-19 20:54:23,766 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1912): c7ee8087efe9c3ae02af767e46864c85/C is initiating minor compaction (all files) 2023-04-19 20:54:23,766 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c7ee8087efe9c3ae02af767e46864c85/C in TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:23,766 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/7334fe85469e4371908bd37f2ae522e1, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/69aa733f9b70435ab3b80dfd86dcdbba, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/7e79e56e618c487f959113e68f83d37d, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/639d44ae22504a589807fcadc1e8e230, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/e01e4a7c480242428ae5667c87c4b5d7, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/b963dd73018b49d78c8e3816f6e38fd1, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/6706bc3aa4054b93b3a4d6dec1537c3f] into tmpdir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp, totalSize=81.4 K 2023-04-19 20:54:23,767 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 7334fe85469e4371908bd37f2ae522e1, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=18, earliestPutTs=1681937657135 2023-04-19 20:54:23,767 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 69aa733f9b70435ab3b80dfd86dcdbba, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=37, earliestPutTs=1681937657162 2023-04-19 20:54:23,768 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 7e79e56e618c487f959113e68f83d37d, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=51, earliestPutTs=1681937657495 2023-04-19 20:54:23,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:54:23,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:54:23,768 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:54:23,768 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:23,768 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:54:23,768 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:54:23,768 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 639d44ae22504a589807fcadc1e8e230, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=60, earliestPutTs=1681937659353 2023-04-19 20:54:23,768 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:23,769 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:23,769 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting e01e4a7c480242428ae5667c87c4b5d7, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=88, earliestPutTs=1681937659429 2023-04-19 20:54:23,769 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting b963dd73018b49d78c8e3816f6e38fd1, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=97, earliestPutTs=1681937660552 2023-04-19 20:54:23,769 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 6706bc3aa4054b93b3a4d6dec1537c3f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=137, earliestPutTs=1681937661696 2023-04-19 20:54:23,770 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:23,770 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:23,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:23,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:23,770 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:23,770 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:54:23,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:23,771 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:23,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:54:23,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:23,771 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:54:23,772 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:54:23,772 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:54:23,772 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:54:23,772 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:23,772 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:54:23,780 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:54:23,780 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:54:23,780 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:23,780 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. started... 2023-04-19 20:54:23,780 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c7ee8087efe9c3ae02af767e46864c85 3/3 column families, dataSize=154.31 KB heapSize=426 KB 2023-04-19 20:54:23,781 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c7ee8087efe9c3ae02af767e46864c85, store=A 2023-04-19 20:54:23,781 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 20:54:23,781 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c7ee8087efe9c3ae02af767e46864c85, store=B 2023-04-19 20:54:23,781 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 20:54:23,781 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c7ee8087efe9c3ae02af767e46864c85, store=C 2023-04-19 20:54:23,782 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 20:54:23,787 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] throttle.PressureAwareThroughputController(145): c7ee8087efe9c3ae02af767e46864c85#C#compaction#128 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 20:54:23,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(9158): Flush requested on c7ee8087efe9c3ae02af767e46864c85 2023-04-19 20:54:23,994 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. as already flushing 2023-04-19 20:54:24,003 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:24,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56352 deadline: 1681937724001, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:24,007 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:24,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56326 deadline: 1681937724002, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:24,008 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:24,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56368 deadline: 1681937724002, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:24,008 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:24,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56408 deadline: 1681937724003, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:24,008 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:24,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56384 deadline: 1681937724003, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:24,105 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:24,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56352 deadline: 1681937724104, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:24,113 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:24,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56326 deadline: 1681937724109, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:24,113 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:24,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56368 deadline: 1681937724109, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:24,113 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:24,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56408 deadline: 1681937724109, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:24,113 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:24,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56384 deadline: 1681937724109, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:24,209 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/C/3ad89c0678f84aed886a6a97bf1d56c6 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/3ad89c0678f84aed886a6a97bf1d56c6 2023-04-19 20:54:24,214 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304194e38d567d6e84913b2328030664a1953 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304194e38d567d6e84913b2328030664a1953 2023-04-19 20:54:24,218 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in c7ee8087efe9c3ae02af767e46864c85/C of c7ee8087efe9c3ae02af767e46864c85 into 3ad89c0678f84aed886a6a97bf1d56c6(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 20:54:24,218 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c7ee8087efe9c3ae02af767e46864c85: 2023-04-19 20:54:24,218 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85., storeName=c7ee8087efe9c3ae02af767e46864c85/C, priority=9, startTime=1681937663696; duration=0sec 2023-04-19 20:54:24,218 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 20:54:24,310 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:24,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56352 deadline: 1681937724307, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:24,315 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:24,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56326 deadline: 1681937724314, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:24,315 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:24,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56408 deadline: 1681937724314, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:24,317 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:24,317 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:24,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56368 deadline: 1681937724315, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:24,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56384 deadline: 1681937724315, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:24,595 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/A/bb2472d2ab864d8fa9f667fb3aa425f0 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/bb2472d2ab864d8fa9f667fb3aa425f0 2023-04-19 20:54:24,601 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in c7ee8087efe9c3ae02af767e46864c85/A of c7ee8087efe9c3ae02af767e46864c85 into bb2472d2ab864d8fa9f667fb3aa425f0(size=25.5 K), total size for store is 25.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 20:54:24,601 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c7ee8087efe9c3ae02af767e46864c85: 2023-04-19 20:54:24,601 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85., storeName=c7ee8087efe9c3ae02af767e46864c85/A, priority=9, startTime=1681937663696; duration=0sec 2023-04-19 20:54:24,601 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 20:54:24,613 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:24,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56352 deadline: 1681937724612, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:24,618 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:24,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56326 deadline: 1681937724618, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:24,619 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:24,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56408 deadline: 1681937724618, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:24,627 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:24,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56384 deadline: 1681937724622, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:24,627 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:24,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56368 deadline: 1681937724623, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:24,631 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=174, memsize=51.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/A/2be537b2a6874d74af6a8dcc5d336676 2023-04-19 20:54:25,055 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=174 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/B/4d84447e502d46c493e355d443136aa5 2023-04-19 20:54:25,118 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:25,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56352 deadline: 1681937725116, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:25,123 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:25,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56408 deadline: 1681937725123, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:25,124 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:25,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56326 deadline: 1681937725124, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:25,129 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:25,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56384 deadline: 1681937725128, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:25,129 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:25,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56368 deadline: 1681937725128, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:25,467 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=174 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/C/73b24293d2364391b6a486932f8b0a30 2023-04-19 20:54:25,472 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/A/2be537b2a6874d74af6a8dcc5d336676 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/2be537b2a6874d74af6a8dcc5d336676 2023-04-19 20:54:25,477 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/2be537b2a6874d74af6a8dcc5d336676, entries=150, sequenceid=174, filesize=25.3 K 2023-04-19 20:54:25,478 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/B/4d84447e502d46c493e355d443136aa5 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/4d84447e502d46c493e355d443136aa5 2023-04-19 20:54:25,482 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/4d84447e502d46c493e355d443136aa5, entries=150, sequenceid=174, filesize=11.8 K 2023-04-19 20:54:25,483 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/C/73b24293d2364391b6a486932f8b0a30 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/73b24293d2364391b6a486932f8b0a30 2023-04-19 20:54:25,488 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/73b24293d2364391b6a486932f8b0a30, entries=150, sequenceid=174, filesize=11.8 K 2023-04-19 20:54:25,489 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~154.31 KB/158010, heapSize ~425.95 KB/436176, currentSize=40.25 KB/41220 for c7ee8087efe9c3ae02af767e46864c85 in 1709ms, sequenceid=174, compaction requested=false 2023-04-19 20:54:25,489 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c7ee8087efe9c3ae02af767e46864c85: 2023-04-19 20:54:25,489 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:25,489 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:54:25,489 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:54:25,489 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:25,489 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:54:25,489 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:54:25,491 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:25,491 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:25,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:25,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:25,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:25,491 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:25,491 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:54:25,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:25,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:25,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:25,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:25,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:25,493 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:54:25,493 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@393fd433[Count = 0] remaining members to acquire global barrier 2023-04-19 20:54:25,493 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:54:25,493 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:25,494 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:25,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:25,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:25,494 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:54:25,494 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:54:25,494 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:54:25,494 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:25,494 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:54:25,497 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:25,497 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:54:25,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:25,497 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:25,497 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:54:25,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:25,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:25,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:25,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:25,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:25,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:25,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:25,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:25,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:25,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:54:25,500 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:54:25,500 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:54:25,500 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:54:25,500 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:54:25,500 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:54:25,500 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:54:25,502 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:25,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:25,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:25,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:25,502 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:25,502 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:25,502 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:25,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:25,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:25,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:25,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:25,503 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:25,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:25,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:25,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:25,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:25,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:25,505 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:25,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:25,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:25,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:25,507 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:25,507 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:25,507 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:25,507 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:54:25,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:25,507 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:25,507 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:54:25,507 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:54:25,507 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:25,507 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:25,507 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:25,507 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:25,507 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:25,508 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:25,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:25,508 DEBUG [Thread-837] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:54:25,508 DEBUG [Thread-837] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:54:25,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:25,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:25,608 DEBUG [Thread-837] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:54:25,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:54:25,610 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:54:25,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:54:25,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:54:25,611 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:54:25,611 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:25,612 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:54:25,612 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:54:25,612 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:25,612 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:25,613 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:25,613 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:25,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:25,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:25,614 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:25,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:25,614 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:54:25,614 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:25,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:54:25,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:25,615 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:54:25,615 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:54:25,615 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:54:25,615 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:25,615 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:54:25,615 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:54:25,615 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:54:25,615 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:25,615 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. started... 2023-04-19 20:54:25,615 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c7ee8087efe9c3ae02af767e46864c85 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-19 20:54:25,616 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c7ee8087efe9c3ae02af767e46864c85, store=A 2023-04-19 20:54:25,616 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:54:25,616 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c7ee8087efe9c3ae02af767e46864c85, store=B 2023-04-19 20:54:25,616 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:54:25,616 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c7ee8087efe9c3ae02af767e46864c85, store=C 2023-04-19 20:54:25,616 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:54:25,641 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041925466cd5005a44578e9be0c7402867c6 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041925466cd5005a44578e9be0c7402867c6 2023-04-19 20:54:25,654 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=185, memsize=13.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/A/c2a34e3cf7504fd28de34777ca22d66f 2023-04-19 20:54:26,070 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=185 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/B/293e606b965d486394b4cc17d9606e69 2023-04-19 20:54:26,081 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=185 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/C/0392f2d555904ac6a78d7a71643261ab 2023-04-19 20:54:26,086 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/A/c2a34e3cf7504fd28de34777ca22d66f as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/c2a34e3cf7504fd28de34777ca22d66f 2023-04-19 20:54:26,090 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/c2a34e3cf7504fd28de34777ca22d66f, entries=150, sequenceid=185, filesize=25.3 K 2023-04-19 20:54:26,091 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/B/293e606b965d486394b4cc17d9606e69 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/293e606b965d486394b4cc17d9606e69 2023-04-19 20:54:26,095 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/293e606b965d486394b4cc17d9606e69, entries=150, sequenceid=185, filesize=11.8 K 2023-04-19 20:54:26,096 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/C/0392f2d555904ac6a78d7a71643261ab as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/0392f2d555904ac6a78d7a71643261ab 2023-04-19 20:54:26,113 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/0392f2d555904ac6a78d7a71643261ab, entries=150, sequenceid=185, filesize=11.8 K 2023-04-19 20:54:26,115 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=0 B/0 for c7ee8087efe9c3ae02af767e46864c85 in 499ms, sequenceid=185, compaction requested=true 2023-04-19 20:54:26,115 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c7ee8087efe9c3ae02af767e46864c85: 2023-04-19 20:54:26,115 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:26,115 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:54:26,115 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:54:26,115 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:26,115 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:54:26,115 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:54:26,118 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:26,118 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:26,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:26,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:26,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:26,118 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:26,119 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:54:26,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:26,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:26,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:26,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:26,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:26,120 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:54:26,120 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@75ce265d[Count = 0] remaining members to acquire global barrier 2023-04-19 20:54:26,120 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:54:26,121 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:26,122 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:26,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:26,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:26,122 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:54:26,122 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:54:26,122 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:54:26,122 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:26,122 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:54:26,124 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:26,124 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:54:26,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:26,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:26,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:26,124 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:26,124 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:54:26,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:26,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:26,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:26,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:26,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:26,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:26,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:26,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:54:26,127 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:54:26,127 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:54:26,127 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:54:26,128 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:54:26,128 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:54:26,128 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:54:26,133 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:26,133 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:26,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:26,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:26,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:26,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:26,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:26,133 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:26,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:26,134 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:26,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:26,134 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:26,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:26,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:26,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:26,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:26,137 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:26,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:26,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:26,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:26,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:26,140 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:26,140 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:26,140 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:54:26,140 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:54:26,140 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:26,140 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:26,140 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:26,140 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:26,140 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:26,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:26,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:26,140 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:26,140 DEBUG [Thread-837] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:54:26,140 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:26,140 DEBUG [Thread-837] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:54:26,140 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:26,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:26,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:26,141 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:54:26,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(9158): Flush requested on c7ee8087efe9c3ae02af767e46864c85 2023-04-19 20:54:26,143 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c7ee8087efe9c3ae02af767e46864c85 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-19 20:54:26,143 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c7ee8087efe9c3ae02af767e46864c85, store=A 2023-04-19 20:54:26,143 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:54:26,143 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c7ee8087efe9c3ae02af767e46864c85, store=B 2023-04-19 20:54:26,143 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:54:26,143 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c7ee8087efe9c3ae02af767e46864c85, store=C 2023-04-19 20:54:26,143 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:54:26,167 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230419ec58f7bd9f0c4fc988f2c37cf3cbbc3f to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419ec58f7bd9f0c4fc988f2c37cf3cbbc3f 2023-04-19 20:54:26,179 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:26,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56352 deadline: 1681937726171, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:26,180 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:26,180 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=859, cellSize=39335, totalHeapSize=89576, min timestamp=1681937666145, max timestamp=1681937666178Num uniques 859; 2023-04-19 20:54:26,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56368 deadline: 1681937726173, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:26,180 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=39335, suffix heap size=117064,new segment heap size=89576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=859 2023-04-19 20:54:26,180 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=859, cellSize=39334, totalHeapSize=89576, min timestamp=1681937666145, max timestamp=1681937666178Num uniques 859; 2023-04-19 20:54:26,181 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=39334, suffix heap size=117064,new segment heap size=89576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=859 2023-04-19 20:54:26,181 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=859, cellSize=39335, totalHeapSize=89576, min timestamp=1681937666145, max timestamp=1681937666178Num uniques 859; 2023-04-19 20:54:26,181 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=39335, suffix heap size=117064,new segment heap size=89576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=859 2023-04-19 20:54:26,190 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=196, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/A/0d2ba020b78d4ed78d1adb713eb8d623 2023-04-19 20:54:26,193 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:26,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56408 deadline: 1681937726189, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:26,197 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:26,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56326 deadline: 1681937726191, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:26,197 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:26,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56384 deadline: 1681937726192, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:26,215 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=196 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/B/b2019f9006c749b5b60c5a36b4338c1e 2023-04-19 20:54:26,240 DEBUG [Thread-837] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:54:26,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:54:26,242 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:54:26,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:54:26,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:54:26,244 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:54:26,244 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:26,244 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:54:26,244 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:54:26,244 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:26,244 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:26,246 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:26,246 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:26,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:26,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:26,246 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:26,246 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:54:26,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:26,246 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:26,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:54:26,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:26,247 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:54:26,247 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:54:26,247 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:54:26,247 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:54:26,247 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:26,247 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:54:26,247 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:54:26,247 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:54:26,247 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:26,247 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. started... 2023-04-19 20:54:26,247 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. as already flushing 2023-04-19 20:54:26,247 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:26,247 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:54:26,248 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:54:26,248 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:26,248 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:54:26,248 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:54:26,249 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:26,249 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:26,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:26,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:26,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:26,249 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:26,249 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:54:26,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:26,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:26,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:26,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:26,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:26,251 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:54:26,251 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@574a3fd8[Count = 0] remaining members to acquire global barrier 2023-04-19 20:54:26,251 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:54:26,251 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:26,253 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:26,253 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:26,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:26,254 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:54:26,254 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:54:26,254 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:54:26,254 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:26,254 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:54:26,255 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:26,255 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:54:26,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:26,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:26,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:26,255 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:26,255 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:54:26,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:26,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:26,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:26,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:26,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:26,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:26,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:26,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:54:26,257 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:54:26,257 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:54:26,257 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:54:26,257 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:54:26,257 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:54:26,257 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:54:26,259 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:26,259 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:26,259 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:26,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:26,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:26,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:26,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:26,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:26,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:26,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:26,259 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:26,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:26,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:26,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:26,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:26,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:26,261 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:26,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:26,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:26,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:26,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:26,263 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:26,263 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:26,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:26,263 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:26,263 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:54:26,263 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:54:26,263 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:26,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:26,263 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:26,263 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:26,263 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:26,263 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:26,263 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:26,263 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:26,263 DEBUG [Thread-837] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:54:26,263 DEBUG [Thread-837] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:54:26,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:26,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:26,284 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:26,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56352 deadline: 1681937726281, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:26,284 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:26,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56368 deadline: 1681937726281, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:26,298 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:26,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56408 deadline: 1681937726294, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:26,299 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:26,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56384 deadline: 1681937726298, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:26,299 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:26,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56326 deadline: 1681937726298, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:26,363 DEBUG [Thread-837] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:54:26,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:54:26,365 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:54:26,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:54:26,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:54:26,367 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:54:26,367 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:26,367 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:54:26,367 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:54:26,367 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:26,367 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:26,370 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:26,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:26,370 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:26,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:26,370 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:26,370 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:54:26,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:26,370 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:26,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:54:26,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:26,371 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:54:26,371 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:54:26,371 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:54:26,371 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:54:26,371 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:26,371 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:54:26,371 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:54:26,371 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:54:26,371 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:26,371 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. started... 2023-04-19 20:54:26,372 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. as already flushing 2023-04-19 20:54:26,372 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:26,372 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:54:26,372 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:54:26,372 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:26,372 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:54:26,372 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:54:26,373 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:26,373 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:26,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:26,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:26,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:26,373 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:26,374 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:54:26,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:26,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:26,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:26,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:26,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:26,375 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:54:26,375 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@302468b1[Count = 0] remaining members to acquire global barrier 2023-04-19 20:54:26,375 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:54:26,375 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:26,376 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:26,376 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:26,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:26,376 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:54:26,376 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:54:26,376 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:54:26,376 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:26,376 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:54:26,377 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:26,377 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:54:26,378 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:26,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:26,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:26,378 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:54:26,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:26,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:26,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:26,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:26,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:26,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:26,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:26,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:26,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:54:26,380 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:54:26,380 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:54:26,380 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:54:26,380 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:54:26,380 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:54:26,380 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:54:26,382 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:26,382 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:26,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:26,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:26,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:26,382 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:26,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:26,382 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:26,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:26,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:26,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:26,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:26,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:26,383 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:26,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:26,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:26,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:26,384 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:26,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:26,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:26,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:26,386 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:26,387 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:26,387 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:26,387 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:26,387 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:26,387 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:26,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:26,387 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:26,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:26,387 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:54:26,387 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:54:26,387 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:26,387 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:26,387 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:26,387 DEBUG [Thread-837] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:54:26,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:26,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:26,387 DEBUG [Thread-837] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:54:26,486 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:26,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56352 deadline: 1681937726485, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:26,486 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:26,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56368 deadline: 1681937726485, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:26,487 DEBUG [Thread-837] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:54:26,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:54:26,490 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:54:26,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:54:26,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:54:26,491 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:54:26,491 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:26,491 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:54:26,491 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:54:26,491 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:26,491 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:26,493 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:26,493 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:26,493 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:26,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:26,493 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:26,493 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:54:26,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:26,494 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:26,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:54:26,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:26,494 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:54:26,494 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:54:26,494 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:54:26,494 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:54:26,495 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:26,495 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:54:26,495 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:54:26,495 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:54:26,495 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:26,495 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. started... 2023-04-19 20:54:26,495 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. as already flushing 2023-04-19 20:54:26,495 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:26,495 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:54:26,495 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:54:26,495 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:26,495 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:54:26,495 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:54:26,498 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:26,498 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:26,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:26,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:26,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:26,498 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:26,498 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:54:26,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:26,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:26,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:26,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:26,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:26,499 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:54:26,500 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@77fdacca[Count = 0] remaining members to acquire global barrier 2023-04-19 20:54:26,500 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:54:26,500 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:26,502 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:26,502 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:26,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:26,502 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:54:26,502 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:54:26,502 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:54:26,502 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:26,502 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:54:26,503 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:26,503 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:54:26,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56408 deadline: 1681937726500, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:26,503 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:26,503 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:26,504 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:26,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56384 deadline: 1681937726501, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:26,504 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:54:26,504 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:26,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56326 deadline: 1681937726501, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:26,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:26,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:26,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:26,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:26,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:26,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:26,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:26,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:26,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:26,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:26,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:54:26,507 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:54:26,507 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:54:26,507 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:54:26,507 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:54:26,507 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:54:26,507 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:54:26,508 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:26,508 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:26,509 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:26,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:26,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:26,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:26,509 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:26,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:26,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:26,509 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:26,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:26,509 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:26,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:26,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:26,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:26,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:26,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:26,511 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:26,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:26,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:26,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:26,516 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:26,516 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:26,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:26,516 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:26,516 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:54:26,516 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:54:26,516 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:54:26,516 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:26,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:26,516 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:26,517 DEBUG [Thread-837] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:54:26,517 DEBUG [Thread-837] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:54:26,517 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:26,517 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:26,517 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:26,517 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:26,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:26,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:26,517 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:26,617 DEBUG [Thread-837] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:54:26,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:54:26,619 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:54:26,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:54:26,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:54:26,620 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:54:26,620 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:26,620 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:54:26,620 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:54:26,621 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:26,621 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:26,622 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:26,622 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:26,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:26,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:26,622 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:26,622 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:54:26,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:26,623 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:26,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:54:26,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:26,623 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:54:26,625 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:54:26,626 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:54:26,626 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:54:26,626 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:26,626 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:54:26,626 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:54:26,626 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:54:26,626 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:26,626 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. started... 2023-04-19 20:54:26,626 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. as already flushing 2023-04-19 20:54:26,626 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:26,626 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:54:26,626 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:54:26,626 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:26,626 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:54:26,626 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:54:26,628 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:26,628 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:26,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:26,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:26,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:26,628 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:26,628 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:54:26,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:26,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:26,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:26,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:26,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:26,629 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:54:26,630 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:54:26,630 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@738580a6[Count = 0] remaining members to acquire global barrier 2023-04-19 20:54:26,630 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:26,631 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:26,631 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:26,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:26,631 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:54:26,631 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:54:26,631 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:54:26,631 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:26,631 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:54:26,632 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=196 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/C/82ed735294aa4455b95553ef97020bab 2023-04-19 20:54:26,634 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:26,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:26,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:26,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:26,635 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:54:26,635 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:26,635 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:54:26,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:26,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:26,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:26,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:26,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:26,637 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/A/0d2ba020b78d4ed78d1adb713eb8d623 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/0d2ba020b78d4ed78d1adb713eb8d623 2023-04-19 20:54:26,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:26,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:26,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:54:26,638 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:54:26,638 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:54:26,638 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:54:26,638 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:54:26,638 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:54:26,638 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:54:26,640 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:26,640 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:26,641 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:26,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:26,641 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:26,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:26,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:26,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:26,641 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:26,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:26,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:26,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:26,642 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/0d2ba020b78d4ed78d1adb713eb8d623, entries=400, sequenceid=196, filesize=59.4 K 2023-04-19 20:54:26,642 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:26,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:26,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:26,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:26,643 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/B/b2019f9006c749b5b60c5a36b4338c1e as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/b2019f9006c749b5b60c5a36b4338c1e 2023-04-19 20:54:26,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:26,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:26,644 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:26,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:26,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:26,646 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:26,646 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:26,646 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:26,646 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:26,646 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:26,646 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:26,647 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:26,647 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:54:26,647 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:54:26,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:26,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:26,646 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:26,647 DEBUG [Thread-837] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:54:26,647 DEBUG [Thread-837] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:54:26,647 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:26,647 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:26,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:26,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:26,648 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/b2019f9006c749b5b60c5a36b4338c1e, entries=400, sequenceid=196, filesize=23.4 K 2023-04-19 20:54:26,649 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/C/82ed735294aa4455b95553ef97020bab as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/82ed735294aa4455b95553ef97020bab 2023-04-19 20:54:26,653 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/82ed735294aa4455b95553ef97020bab, entries=400, sequenceid=196, filesize=23.4 K 2023-04-19 20:54:26,654 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=147.60 KB/151140 for c7ee8087efe9c3ae02af767e46864c85 in 512ms, sequenceid=196, compaction requested=true 2023-04-19 20:54:26,654 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c7ee8087efe9c3ae02af767e46864c85: 2023-04-19 20:54:26,654 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 20:54:26,654 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-19 20:54:26,654 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-19 20:54:26,654 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 20:54:26,655 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 20:54:26,656 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 138782 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-19 20:54:26,656 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1912): c7ee8087efe9c3ae02af767e46864c85/A is initiating minor compaction (all files) 2023-04-19 20:54:26,656 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c7ee8087efe9c3ae02af767e46864c85/A in TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:26,656 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 60338 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-19 20:54:26,656 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/bb2472d2ab864d8fa9f667fb3aa425f0, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/2be537b2a6874d74af6a8dcc5d336676, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/c2a34e3cf7504fd28de34777ca22d66f, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/0d2ba020b78d4ed78d1adb713eb8d623] into tmpdir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp, totalSize=135.5 K 2023-04-19 20:54:26,656 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1912): c7ee8087efe9c3ae02af767e46864c85/B is initiating minor compaction (all files) 2023-04-19 20:54:26,656 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c7ee8087efe9c3ae02af767e46864c85/B in TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:26,656 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/e944bb05f5534a5fb5d3caa0bdea0e1e, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/4d84447e502d46c493e355d443136aa5, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/293e606b965d486394b4cc17d9606e69, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/b2019f9006c749b5b60c5a36b4338c1e] into tmpdir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp, totalSize=58.9 K 2023-04-19 20:54:26,657 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting bb2472d2ab864d8fa9f667fb3aa425f0, keycount=150, bloomtype=ROW, size=25.5 K, encoding=NONE, compression=NONE, seqNum=137, earliestPutTs=1681937661696 2023-04-19 20:54:26,657 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting e944bb05f5534a5fb5d3caa0bdea0e1e, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=137, earliestPutTs=1681937661696 2023-04-19 20:54:26,657 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 2be537b2a6874d74af6a8dcc5d336676, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=174, earliestPutTs=1681937662863 2023-04-19 20:54:26,658 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 4d84447e502d46c493e355d443136aa5, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=174, earliestPutTs=1681937662863 2023-04-19 20:54:26,658 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting c2a34e3cf7504fd28de34777ca22d66f, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=185, earliestPutTs=1681937663993 2023-04-19 20:54:26,658 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 293e606b965d486394b4cc17d9606e69, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=185, earliestPutTs=1681937663993 2023-04-19 20:54:26,659 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting b2019f9006c749b5b60c5a36b4338c1e, keycount=400, bloomtype=ROW, size=23.4 K, encoding=NONE, compression=NONE, seqNum=196, earliestPutTs=1681937666132 2023-04-19 20:54:26,662 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 0d2ba020b78d4ed78d1adb713eb8d623, keycount=400, bloomtype=ROW, size=59.4 K, encoding=NONE, compression=NONE, seqNum=196, earliestPutTs=1681937666132 2023-04-19 20:54:26,690 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] throttle.PressureAwareThroughputController(145): c7ee8087efe9c3ae02af767e46864c85#B#compaction#138 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 20:54:26,702 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] throttle.PressureAwareThroughputController(145): c7ee8087efe9c3ae02af767e46864c85#A#compaction#139 average throughput is 1.79 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 20:54:26,747 DEBUG [Thread-837] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:54:26,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:54:26,749 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:54:26,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:54:26,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:54:26,750 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:54:26,750 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:26,750 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:54:26,750 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:54:26,751 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:26,751 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:26,752 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:26,753 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:26,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:26,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:26,753 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:26,753 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:54:26,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:26,753 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:26,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:54:26,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:26,754 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:54:26,754 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:54:26,754 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:54:26,754 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:54:26,754 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:26,754 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:54:26,760 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:54:26,760 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:54:26,760 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:26,760 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. started... 2023-04-19 20:54:26,760 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c7ee8087efe9c3ae02af767e46864c85 3/3 column families, dataSize=147.60 KB heapSize=359.20 KB 2023-04-19 20:54:26,760 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c7ee8087efe9c3ae02af767e46864c85, store=A 2023-04-19 20:54:26,760 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 20:54:26,760 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c7ee8087efe9c3ae02af767e46864c85, store=B 2023-04-19 20:54:26,760 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 20:54:26,761 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c7ee8087efe9c3ae02af767e46864c85, store=C 2023-04-19 20:54:26,761 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 20:54:26,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(9158): Flush requested on c7ee8087efe9c3ae02af767e46864c85 2023-04-19 20:54:26,790 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. as already flushing 2023-04-19 20:54:26,804 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:26,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56368 deadline: 1681937726803, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:26,804 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:26,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56352 deadline: 1681937726803, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:26,810 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:26,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56408 deadline: 1681937726806, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:26,810 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:26,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56384 deadline: 1681937726807, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:26,810 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:26,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56326 deadline: 1681937726807, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:26,908 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:26,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56368 deadline: 1681937726905, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:26,909 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:26,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56352 deadline: 1681937726905, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:27,113 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:27,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56352 deadline: 1681937727111, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:27,113 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:27,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56368 deadline: 1681937727111, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:27,121 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/B/efcb2c3696da4e81a524a316e83b9536 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/efcb2c3696da4e81a524a316e83b9536 2023-04-19 20:54:27,135 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in c7ee8087efe9c3ae02af767e46864c85/B of c7ee8087efe9c3ae02af767e46864c85 into efcb2c3696da4e81a524a316e83b9536(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 20:54:27,135 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c7ee8087efe9c3ae02af767e46864c85: 2023-04-19 20:54:27,135 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85., storeName=c7ee8087efe9c3ae02af767e46864c85/B, priority=12, startTime=1681937666654; duration=0sec 2023-04-19 20:54:27,135 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 20:54:27,135 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-19 20:54:27,137 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 60338 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-19 20:54:27,137 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1912): c7ee8087efe9c3ae02af767e46864c85/C is initiating minor compaction (all files) 2023-04-19 20:54:27,137 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c7ee8087efe9c3ae02af767e46864c85/C in TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:27,137 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/3ad89c0678f84aed886a6a97bf1d56c6, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/73b24293d2364391b6a486932f8b0a30, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/0392f2d555904ac6a78d7a71643261ab, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/82ed735294aa4455b95553ef97020bab] into tmpdir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp, totalSize=58.9 K 2023-04-19 20:54:27,138 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 3ad89c0678f84aed886a6a97bf1d56c6, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=137, earliestPutTs=1681937661696 2023-04-19 20:54:27,139 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 73b24293d2364391b6a486932f8b0a30, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=174, earliestPutTs=1681937662863 2023-04-19 20:54:27,139 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 0392f2d555904ac6a78d7a71643261ab, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=185, earliestPutTs=1681937663993 2023-04-19 20:54:27,139 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 82ed735294aa4455b95553ef97020bab, keycount=400, bloomtype=ROW, size=23.4 K, encoding=NONE, compression=NONE, seqNum=196, earliestPutTs=1681937666132 2023-04-19 20:54:27,149 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] throttle.PressureAwareThroughputController(145): c7ee8087efe9c3ae02af767e46864c85#C#compaction#141 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 20:54:27,162 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/C/dbcb253d0c6446ac9d7b512ae1f30263 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/dbcb253d0c6446ac9d7b512ae1f30263 2023-04-19 20:54:27,169 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in c7ee8087efe9c3ae02af767e46864c85/C of c7ee8087efe9c3ae02af767e46864c85 into dbcb253d0c6446ac9d7b512ae1f30263(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 20:54:27,169 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c7ee8087efe9c3ae02af767e46864c85: 2023-04-19 20:54:27,169 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85., storeName=c7ee8087efe9c3ae02af767e46864c85/C, priority=12, startTime=1681937666655; duration=0sec 2023-04-19 20:54:27,169 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 20:54:27,174 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230419595b72847c864ff2a1fad3fbd031f50f to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419595b72847c864ff2a1fad3fbd031f50f 2023-04-19 20:54:27,312 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:27,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56408 deadline: 1681937727311, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:27,315 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:27,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56384 deadline: 1681937727314, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:27,317 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:27,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56326 deadline: 1681937727315, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:27,416 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:27,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56352 deadline: 1681937727414, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:27,416 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:27,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56368 deadline: 1681937727415, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:27,556 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/A/453272501ae24401b5450a3a6abca5da as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/453272501ae24401b5450a3a6abca5da 2023-04-19 20:54:27,561 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in c7ee8087efe9c3ae02af767e46864c85/A of c7ee8087efe9c3ae02af767e46864c85 into 453272501ae24401b5450a3a6abca5da(size=25.7 K), total size for store is 25.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 20:54:27,561 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c7ee8087efe9c3ae02af767e46864c85: 2023-04-19 20:54:27,561 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85., storeName=c7ee8087efe9c3ae02af767e46864c85/A, priority=12, startTime=1681937666654; duration=0sec 2023-04-19 20:54:27,561 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 20:54:27,582 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=221, memsize=49.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/A/536796bf8ff64c8aaef8f2c36c3939ca 2023-04-19 20:54:27,920 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:27,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56368 deadline: 1681937727918, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:27,921 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:27,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56352 deadline: 1681937727920, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:27,993 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=221 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/B/6d3341a472ae4b2896debd09305e0c4a 2023-04-19 20:54:28,318 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:28,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56326 deadline: 1681937728318, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:28,322 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:28,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56408 deadline: 1681937728320, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:28,326 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:28,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56384 deadline: 1681937728325, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:28,411 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=221 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/C/469dabb7d280474eadb90a7abfc38076 2023-04-19 20:54:28,416 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/A/536796bf8ff64c8aaef8f2c36c3939ca as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/536796bf8ff64c8aaef8f2c36c3939ca 2023-04-19 20:54:28,420 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/536796bf8ff64c8aaef8f2c36c3939ca, entries=150, sequenceid=221, filesize=25.3 K 2023-04-19 20:54:28,421 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/B/6d3341a472ae4b2896debd09305e0c4a as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/6d3341a472ae4b2896debd09305e0c4a 2023-04-19 20:54:28,427 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/6d3341a472ae4b2896debd09305e0c4a, entries=150, sequenceid=221, filesize=11.8 K 2023-04-19 20:54:28,428 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/C/469dabb7d280474eadb90a7abfc38076 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/469dabb7d280474eadb90a7abfc38076 2023-04-19 20:54:28,432 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/469dabb7d280474eadb90a7abfc38076, entries=150, sequenceid=221, filesize=11.8 K 2023-04-19 20:54:28,433 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~359.16 KB/367776, currentSize=53.67 KB/54960 for c7ee8087efe9c3ae02af767e46864c85 in 1673ms, sequenceid=221, compaction requested=false 2023-04-19 20:54:28,433 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c7ee8087efe9c3ae02af767e46864c85: 2023-04-19 20:54:28,433 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:28,434 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:54:28,434 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:54:28,434 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:28,434 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:54:28,434 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:54:28,436 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:28,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:28,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:28,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:28,437 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:28,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:28,437 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:28,437 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:54:28,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:28,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:28,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:28,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:28,439 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:54:28,439 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@75b7c5f1[Count = 0] remaining members to acquire global barrier 2023-04-19 20:54:28,439 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:54:28,439 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:28,440 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:28,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:28,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:28,440 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:54:28,440 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:54:28,440 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:54:28,440 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:28,440 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:54:28,441 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:54:28,441 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:28,441 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:28,442 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:54:28,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:28,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:28,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:28,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:28,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:28,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:28,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:28,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:28,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:28,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:28,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:54:28,444 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:54:28,444 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:54:28,444 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:54:28,444 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:54:28,444 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:54:28,444 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:54:28,447 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:28,447 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:28,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:28,447 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:28,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:28,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:28,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:28,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:28,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:28,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:28,448 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:28,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:28,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:28,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:28,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:28,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:28,449 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:28,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:28,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:28,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:28,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:28,451 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:28,451 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:28,451 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:28,451 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:28,452 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:28,452 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:28,452 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:28,452 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:54:28,452 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:54:28,452 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:28,452 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:28,452 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:28,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:28,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:28,452 DEBUG [Thread-837] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:54:28,452 DEBUG [Thread-837] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:54:28,452 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:54:28,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:28,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:28,552 DEBUG [Thread-837] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:54:28,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:54:28,555 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:54:28,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:54:28,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:54:28,556 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:54:28,556 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:28,556 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:54:28,556 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:54:28,557 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:28,557 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:28,560 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:28,560 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:28,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:28,560 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:28,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:28,561 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:28,561 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:54:28,561 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:28,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:54:28,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:28,561 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:54:28,562 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:54:28,562 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:54:28,562 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:54:28,562 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:28,562 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:54:28,562 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:54:28,562 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:54:28,562 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:28,562 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. started... 2023-04-19 20:54:28,562 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c7ee8087efe9c3ae02af767e46864c85 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-19 20:54:28,562 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c7ee8087efe9c3ae02af767e46864c85, store=A 2023-04-19 20:54:28,562 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:54:28,562 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c7ee8087efe9c3ae02af767e46864c85, store=B 2023-04-19 20:54:28,562 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:54:28,562 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c7ee8087efe9c3ae02af767e46864c85, store=C 2023-04-19 20:54:28,562 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:54:28,582 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230419bab3c6cfe3a848bcb0bc4ed0139e5a4f to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419bab3c6cfe3a848bcb0bc4ed0139e5a4f 2023-04-19 20:54:28,588 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=235, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/A/e6ed8f4c953c463d894ef3e50a5c1419 2023-04-19 20:54:28,614 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=235 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/B/a19cd3aeeab94e3b829aa79e558ca40f 2023-04-19 20:54:28,631 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=235 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/C/9aa7e84de7f24bd5a24e41acbc9f3709 2023-04-19 20:54:28,635 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/A/e6ed8f4c953c463d894ef3e50a5c1419 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/e6ed8f4c953c463d894ef3e50a5c1419 2023-04-19 20:54:28,639 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/e6ed8f4c953c463d894ef3e50a5c1419, entries=150, sequenceid=235, filesize=25.3 K 2023-04-19 20:54:28,640 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/B/a19cd3aeeab94e3b829aa79e558ca40f as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/a19cd3aeeab94e3b829aa79e558ca40f 2023-04-19 20:54:28,650 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/a19cd3aeeab94e3b829aa79e558ca40f, entries=150, sequenceid=235, filesize=11.8 K 2023-04-19 20:54:28,660 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/C/9aa7e84de7f24bd5a24e41acbc9f3709 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/9aa7e84de7f24bd5a24e41acbc9f3709 2023-04-19 20:54:28,670 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/9aa7e84de7f24bd5a24e41acbc9f3709, entries=150, sequenceid=235, filesize=11.8 K 2023-04-19 20:54:28,671 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=0 B/0 for c7ee8087efe9c3ae02af767e46864c85 in 109ms, sequenceid=235, compaction requested=true 2023-04-19 20:54:28,671 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c7ee8087efe9c3ae02af767e46864c85: 2023-04-19 20:54:28,671 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:28,671 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:54:28,671 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:54:28,671 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:28,671 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:54:28,671 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:54:28,673 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:28,673 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:28,673 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:54:28,674 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:28,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:28,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:28,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:28,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:28,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:28,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:28,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:28,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:28,676 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:54:28,676 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@66455a16[Count = 0] remaining members to acquire global barrier 2023-04-19 20:54:28,676 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:54:28,676 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:28,679 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:28,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:28,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:28,679 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:54:28,679 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:54:28,679 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:54:28,679 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:28,679 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:54:28,680 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:28,680 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:54:28,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:28,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:28,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:28,681 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:28,681 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:54:28,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:28,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:28,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:28,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:28,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:28,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:28,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:28,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:54:28,684 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:54:28,684 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:54:28,684 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:54:28,684 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:54:28,684 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:54:28,684 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:54:28,685 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:28,685 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:28,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:28,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:28,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:28,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:28,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:28,686 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:28,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:28,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:28,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:28,686 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:28,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:28,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:28,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:28,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:28,688 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:28,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:28,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:28,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:28,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:28,693 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:28,693 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:28,693 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:28,693 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:28,693 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:28,693 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:28,693 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:28,693 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:54:28,694 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:54:28,693 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:28,694 DEBUG [Thread-837] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:54:28,694 DEBUG [Thread-837] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:54:28,697 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:28,697 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:28,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:28,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:28,697 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:54:28,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:28,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:28,794 DEBUG [Thread-837] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:54:28,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:54:28,800 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:54:28,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:54:28,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:54:28,801 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:54:28,801 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:28,801 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:54:28,801 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:54:28,802 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:28,802 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:28,803 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:28,803 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:28,803 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:28,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:28,803 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:28,804 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:54:28,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:28,804 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:28,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:54:28,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:28,804 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:54:28,805 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:54:28,805 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:54:28,805 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:54:28,805 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:28,805 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:54:28,805 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:54:28,805 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:54:28,805 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:28,806 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. started... 2023-04-19 20:54:28,806 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c7ee8087efe9c3ae02af767e46864c85: 2023-04-19 20:54:28,806 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:28,806 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:54:28,806 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:54:28,806 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:28,806 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:54:28,806 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:54:28,807 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:28,808 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:28,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:28,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:28,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:28,808 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:28,808 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:54:28,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:28,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:28,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:28,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:28,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:28,810 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:54:28,810 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:54:28,810 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7cc431e8[Count = 0] remaining members to acquire global barrier 2023-04-19 20:54:28,810 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:28,812 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:28,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:28,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:28,812 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:54:28,812 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:54:28,812 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:54:28,812 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:28,812 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:54:28,814 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:28,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:28,814 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:54:28,814 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:28,814 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:54:28,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:28,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:28,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:28,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:28,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:28,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:28,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:28,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:28,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:28,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:54:28,818 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:54:28,818 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:54:28,818 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:54:28,818 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:54:28,818 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:54:28,818 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:54:28,820 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:28,820 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:28,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:28,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:28,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:28,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:28,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:28,820 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:28,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:28,821 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:28,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:28,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:28,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:28,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:28,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:28,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:28,822 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:28,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:28,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:28,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:28,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:28,825 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:28,825 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:28,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:28,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:28,825 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:28,825 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:28,825 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:28,825 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:28,825 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:54:28,825 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:54:28,825 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:28,825 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:28,825 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:28,825 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:28,826 DEBUG [Thread-837] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:54:28,826 DEBUG [Thread-837] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:54:28,826 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:28,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:28,826 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:54:28,926 DEBUG [Thread-837] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:54:28,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:54:28,928 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:54:28,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:54:28,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:54:28,932 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:54:28,932 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:28,932 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:54:28,932 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:54:28,933 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:28,933 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:28,934 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:28,934 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:28,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:28,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:28,934 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:28,935 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:54:28,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:28,935 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:28,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:54:28,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:28,936 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:54:28,936 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:54:28,936 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:54:28,936 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:54:28,936 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:28,936 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:54:28,936 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:54:28,936 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:54:28,936 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:28,936 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. started... 2023-04-19 20:54:28,936 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c7ee8087efe9c3ae02af767e46864c85: 2023-04-19 20:54:28,936 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:28,937 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:54:28,937 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:54:28,937 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:28,937 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:54:28,937 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:54:28,939 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:28,939 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:28,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:28,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:28,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:28,940 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:28,940 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:54:28,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:28,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:28,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:28,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:28,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:28,941 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:54:28,942 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:54:28,942 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@231b0e66[Count = 0] remaining members to acquire global barrier 2023-04-19 20:54:28,942 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:28,943 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:28,943 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:28,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:28,943 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:54:28,943 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:54:28,943 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:54:28,943 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:28,943 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:54:28,944 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:28,944 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:54:28,944 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:28,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:28,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:28,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:28,944 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:54:28,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:28,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:28,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:28,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:28,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:28,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:28,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:28,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:54:28,948 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:54:28,948 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:54:28,948 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:54:28,948 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:54:28,948 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:54:28,948 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:54:28,950 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:28,950 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:28,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:28,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:28,950 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:28,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:28,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:28,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:28,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:28,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:28,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:28,951 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:28,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:28,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:28,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:28,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:28,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:28,952 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:28,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:28,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:28,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:28,956 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:28,956 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:28,956 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:28,956 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:28,956 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:28,956 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:28,956 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:28,956 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:28,956 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:28,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:28,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:28,956 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:28,956 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:54:28,956 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:54:28,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:28,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:28,957 DEBUG [Thread-837] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:54:28,957 DEBUG [Thread-837] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:54:28,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(9158): Flush requested on c7ee8087efe9c3ae02af767e46864c85 2023-04-19 20:54:28,981 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c7ee8087efe9c3ae02af767e46864c85 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-19 20:54:28,981 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c7ee8087efe9c3ae02af767e46864c85, store=A 2023-04-19 20:54:28,982 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:54:28,982 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c7ee8087efe9c3ae02af767e46864c85, store=B 2023-04-19 20:54:28,982 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:54:28,982 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c7ee8087efe9c3ae02af767e46864c85, store=C 2023-04-19 20:54:28,982 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:54:29,019 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:29,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56352 deadline: 1681937729016, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:29,020 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681937669007, max timestamp=1681937669017Num uniques 859; 2023-04-19 20:54:29,020 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 20:54:29,020 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681937669007, max timestamp=1681937669017Num uniques 859; 2023-04-19 20:54:29,020 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 20:54:29,021 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681937669007, max timestamp=1681937669017Num uniques 859; 2023-04-19 20:54:29,021 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 20:54:29,057 DEBUG [Thread-837] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:54:29,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:54:29,059 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:54:29,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:54:29,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:54:29,060 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:54:29,060 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:29,060 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:54:29,060 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:54:29,061 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:29,061 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:29,062 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:29,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:29,062 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:29,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:29,062 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:29,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:29,062 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:54:29,063 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:29,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:54:29,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:29,063 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:54:29,063 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:54:29,064 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:54:29,064 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:54:29,064 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:29,064 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:54:29,064 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:54:29,064 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:54:29,064 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:29,064 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. started... 2023-04-19 20:54:29,064 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. as already flushing 2023-04-19 20:54:29,064 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:29,064 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:54:29,064 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:54:29,064 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:29,064 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:54:29,064 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:54:29,066 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:29,066 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:29,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:29,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:29,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:29,066 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:29,066 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:54:29,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:29,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:29,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:29,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:29,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:29,068 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:54:29,068 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@68c8dfb9[Count = 0] remaining members to acquire global barrier 2023-04-19 20:54:29,068 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:54:29,068 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:29,069 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:29,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:29,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:29,069 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:54:29,069 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:54:29,069 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:54:29,069 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:29,069 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:54:29,071 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:29,071 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:54:29,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:29,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:29,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:29,071 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:29,072 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:54:29,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:29,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:29,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:29,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:29,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:29,076 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:29,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56368 deadline: 1681937729074, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:29,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:29,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:29,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:54:29,079 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:54:29,079 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:54:29,079 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:54:29,079 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:54:29,079 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:54:29,079 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:54:29,081 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:29,081 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:29,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:29,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:29,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:29,081 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:29,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:29,081 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:29,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:29,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:29,082 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:29,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:29,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:29,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:29,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:29,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:29,083 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:29,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:29,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:29,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:29,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:29,086 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:29,086 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:29,086 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:29,086 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:29,086 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:29,086 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:29,086 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:29,086 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:29,086 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:29,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:29,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:29,087 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:29,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:29,087 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:29,087 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:54:29,087 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:54:29,088 DEBUG [Thread-837] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:54:29,088 DEBUG [Thread-837] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:54:29,088 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:54:29,121 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:29,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56352 deadline: 1681937729120, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:29,182 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:29,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56368 deadline: 1681937729178, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:29,188 DEBUG [Thread-837] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:54:29,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:54:29,190 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:54:29,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:54:29,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:54:29,191 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:54:29,191 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:29,191 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:54:29,191 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:54:29,192 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:29,192 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:29,193 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:29,193 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:29,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:29,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:29,194 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:29,194 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:54:29,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:29,194 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:29,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:54:29,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:29,194 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:54:29,194 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:54:29,194 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:54:29,195 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:54:29,195 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:29,195 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:54:29,195 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:54:29,195 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:54:29,195 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:29,195 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. started... 2023-04-19 20:54:29,195 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. as already flushing 2023-04-19 20:54:29,195 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:29,195 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:54:29,195 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:54:29,195 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:29,195 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:54:29,195 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:54:29,201 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:29,201 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:29,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:29,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:29,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:29,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:29,202 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:29,202 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:54:29,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:29,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:29,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:29,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:29,203 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:54:29,203 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2411ad5c[Count = 0] remaining members to acquire global barrier 2023-04-19 20:54:29,203 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:54:29,203 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:29,204 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:29,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:29,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:29,204 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:54:29,204 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:29,204 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:54:29,204 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:54:29,204 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:54:29,206 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:29,206 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:54:29,206 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:29,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:29,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:29,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:29,206 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:54:29,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:29,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:29,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:29,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:29,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:29,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:29,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:29,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:54:29,208 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:54:29,208 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:54:29,208 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:54:29,208 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:54:29,208 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:54:29,208 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:54:29,211 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:29,211 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:29,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:29,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:29,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:29,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:29,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:29,211 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:29,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:29,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:29,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:29,211 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:29,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:29,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:29,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:29,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:29,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:29,212 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:29,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:29,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:29,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:29,214 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:29,214 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:29,214 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:29,214 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:29,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:29,215 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:54:29,214 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:29,215 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:54:29,215 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:29,214 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:29,215 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:29,215 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:29,215 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:29,215 DEBUG [Thread-837] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:54:29,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:29,215 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:29,215 DEBUG [Thread-837] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:54:29,215 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:29,315 DEBUG [Thread-837] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:54:29,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:54:29,317 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:54:29,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:54:29,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:54:29,318 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:54:29,318 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:29,319 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:54:29,319 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:54:29,319 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:29,319 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:29,320 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:29,320 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:29,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:29,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:29,320 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:29,320 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:54:29,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:29,321 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:29,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:54:29,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:29,321 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:54:29,322 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:54:29,322 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:54:29,322 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:54:29,322 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:29,322 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:54:29,322 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:54:29,322 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:54:29,322 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:29,322 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. started... 2023-04-19 20:54:29,322 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. as already flushing 2023-04-19 20:54:29,322 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:29,322 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:54:29,322 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:54:29,322 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:29,322 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:54:29,322 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:54:29,324 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:29,324 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:29,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:29,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:29,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:29,324 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:29,324 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:54:29,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:29,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:29,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:29,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:29,325 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:29,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56352 deadline: 1681937729324, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:29,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:29,326 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:54:29,326 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@79079347[Count = 0] remaining members to acquire global barrier 2023-04-19 20:54:29,326 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:54:29,326 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:29,328 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:29,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:29,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:29,329 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:54:29,329 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:54:29,329 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:54:29,329 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:29,329 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:54:29,330 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:54:29,330 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:29,330 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:29,330 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:54:29,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:29,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:29,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:29,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:29,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:29,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:29,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:29,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:29,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:29,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:29,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:54:29,332 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:54:29,332 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:54:29,333 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:54:29,333 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:54:29,333 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:54:29,333 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:54:29,334 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:29,334 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:29,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:29,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:29,334 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:29,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:29,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:29,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:29,335 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:29,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:29,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:29,335 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:29,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:29,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:29,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:29,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:29,336 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:29,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:29,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:29,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:29,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:29,338 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:29,338 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:29,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:29,338 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:54:29,338 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:54:29,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:29,338 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:29,338 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:29,338 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:29,339 DEBUG [Thread-837] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:54:29,339 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:54:29,339 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:29,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:29,339 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:29,339 DEBUG [Thread-837] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:54:29,339 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:29,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:29,339 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:29,339 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:29,385 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:29,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56368 deadline: 1681937729383, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:29,408 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304197d217cc663c549a9bfa900cd7ab03e4a to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304197d217cc663c549a9bfa900cd7ab03e4a 2023-04-19 20:54:29,439 DEBUG [Thread-837] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:54:29,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:54:29,441 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:54:29,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:54:29,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:54:29,442 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:54:29,442 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:29,442 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:54:29,442 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:54:29,443 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:29,443 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:29,445 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:29,445 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:29,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:29,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:29,445 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:29,446 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:54:29,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:29,446 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:29,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:54:29,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:29,446 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:54:29,446 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:54:29,446 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:54:29,447 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:54:29,447 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:29,447 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:54:29,447 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:54:29,447 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:54:29,447 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:29,447 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. started... 2023-04-19 20:54:29,447 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. as already flushing 2023-04-19 20:54:29,447 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:29,447 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:54:29,447 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:54:29,447 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:29,447 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:54:29,447 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:54:29,449 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:29,449 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:29,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:29,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:29,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:29,449 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:29,449 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:54:29,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:29,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:29,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:29,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:29,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:29,450 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:54:29,450 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@294ca3f[Count = 0] remaining members to acquire global barrier 2023-04-19 20:54:29,450 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:54:29,450 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:29,452 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:29,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:29,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:29,452 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:54:29,452 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:54:29,452 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:54:29,452 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:29,452 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:54:29,453 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:29,453 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:54:29,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:29,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:29,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:29,453 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:29,453 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:54:29,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:29,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:29,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:29,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:29,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:29,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:29,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:29,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:54:29,455 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:54:29,455 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:54:29,455 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:54:29,455 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:54:29,456 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:54:29,456 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:54:29,457 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:29,457 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:29,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:29,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:29,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:29,458 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:29,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:29,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:29,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:29,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:29,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:29,458 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:29,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:29,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:29,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:29,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:29,459 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:29,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:29,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:29,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:29,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:29,462 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:29,462 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:29,462 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:29,462 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:29,462 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:54:29,462 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:54:29,462 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:54:29,462 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:29,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:29,462 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:29,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:29,463 DEBUG [Thread-837] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:54:29,462 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:29,463 DEBUG [Thread-837] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:54:29,463 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:29,463 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:29,463 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:29,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:29,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:29,563 DEBUG [Thread-837] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:54:29,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:54:29,565 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:54:29,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:54:29,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:54:29,566 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:54:29,566 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:29,566 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:54:29,566 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:54:29,567 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:29,567 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:29,568 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:29,568 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:29,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:29,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:29,568 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:29,568 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:54:29,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:29,569 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:29,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:54:29,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:29,569 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:54:29,569 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:54:29,569 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:54:29,569 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:54:29,569 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:29,569 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:54:29,570 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:54:29,570 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:54:29,570 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:29,570 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. started... 2023-04-19 20:54:29,570 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. as already flushing 2023-04-19 20:54:29,570 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:29,570 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:54:29,570 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:54:29,570 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:29,570 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:54:29,570 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:54:29,571 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:29,571 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:29,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:29,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:29,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:29,572 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:29,572 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:54:29,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:29,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:29,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:29,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:29,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:29,573 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:54:29,573 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2c8ff8c3[Count = 0] remaining members to acquire global barrier 2023-04-19 20:54:29,573 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:54:29,573 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:29,578 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:29,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:29,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:29,578 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:29,579 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:54:29,578 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:54:29,579 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:54:29,579 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:54:29,580 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:54:29,580 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:29,580 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:29,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:29,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:29,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:29,580 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:54:29,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:29,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:29,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:29,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:29,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:29,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:29,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:29,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:54:29,589 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:54:29,589 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:54:29,589 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:54:29,589 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:54:29,589 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:54:29,589 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:54:29,591 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:29,591 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:29,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:29,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:29,591 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:29,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:29,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:29,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:29,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:29,592 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:29,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:29,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:29,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:29,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:29,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:29,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:29,593 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:29,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:29,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:29,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:29,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:29,595 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:29,595 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:29,595 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:29,595 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:29,595 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:29,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:29,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:29,596 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:29,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:29,596 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:29,596 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:29,596 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:29,596 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:29,596 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:29,596 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:54:29,596 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:54:29,597 DEBUG [Thread-837] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:54:29,597 DEBUG [Thread-837] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:54:29,597 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:54:29,627 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:29,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56352 deadline: 1681937729626, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:29,689 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:29,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56368 deadline: 1681937729686, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:29,697 DEBUG [Thread-837] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:54:29,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:54:29,699 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:54:29,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:54:29,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:54:29,700 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:54:29,700 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:29,700 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:54:29,700 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:54:29,701 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:29,701 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:29,703 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:29,703 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:29,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:29,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:29,703 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:29,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:29,703 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:54:29,704 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:29,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:54:29,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:29,704 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:54:29,704 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:54:29,704 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:54:29,704 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:54:29,704 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:29,704 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:54:29,705 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:54:29,705 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:54:29,705 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:29,705 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. started... 2023-04-19 20:54:29,705 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. as already flushing 2023-04-19 20:54:29,705 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:29,705 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:54:29,705 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:54:29,705 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:29,705 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:54:29,705 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:54:29,707 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:29,707 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:29,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:29,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:29,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:29,707 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:29,707 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:54:29,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:29,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:29,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:29,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:29,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:29,708 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:54:29,708 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@55ddb43[Count = 0] remaining members to acquire global barrier 2023-04-19 20:54:29,708 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:54:29,709 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:29,710 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:29,710 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:29,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:29,710 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:54:29,710 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:54:29,710 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:54:29,710 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:29,710 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:54:29,711 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:29,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:29,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:29,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:29,712 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:54:29,712 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:29,712 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:54:29,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:29,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:29,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:29,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:29,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:29,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:29,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:29,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:54:29,714 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:54:29,714 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:54:29,714 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:54:29,714 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:54:29,714 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:54:29,714 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:54:29,717 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:29,717 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:29,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:29,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:29,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:29,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:29,717 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:29,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:29,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:29,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:29,717 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:29,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:29,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:29,719 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:29,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:29,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:29,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:29,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:29,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:29,721 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:29,721 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:29,721 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:29,721 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:29,721 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:29,721 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:29,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:29,721 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:54:29,721 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:54:29,721 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:29,722 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:54:29,721 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:29,722 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:29,722 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:29,722 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:29,722 DEBUG [Thread-837] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:54:29,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:29,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:29,722 DEBUG [Thread-837] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:54:29,815 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=248, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/A/1d011e0999b9447392914e99e6e1fbd3 2023-04-19 20:54:29,822 DEBUG [Thread-837] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:54:29,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:54:29,825 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:54:29,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:54:29,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:54:29,833 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=248 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/B/c3be8268e328449f998c7138ee775f6b 2023-04-19 20:54:29,833 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:54:29,834 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:29,834 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:54:29,834 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:54:29,834 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:29,834 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:29,835 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:29,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:29,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:29,836 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:29,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:29,836 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:29,836 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:54:29,837 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:29,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:54:29,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:29,837 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:54:29,838 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:54:29,838 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:54:29,838 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:54:29,838 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:29,838 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:54:29,838 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:54:29,838 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:54:29,838 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:29,838 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. started... 2023-04-19 20:54:29,838 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. as already flushing 2023-04-19 20:54:29,838 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:29,838 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:54:29,838 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:54:29,838 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:29,838 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:54:29,838 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:54:29,840 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:29,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:29,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:29,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:29,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:29,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:29,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:29,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:29,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:29,842 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:29,843 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:54:29,843 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4085ed04[Count = 0] remaining members to acquire global barrier 2023-04-19 20:54:29,843 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:54:29,843 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:29,843 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:54:29,843 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:29,846 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:29,846 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:54:29,847 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:29,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:29,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:29,847 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:54:29,847 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:54:29,847 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:54:29,848 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:54:29,848 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:29,848 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:29,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:29,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:29,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:29,848 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:54:29,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:29,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:29,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:29,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:29,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:29,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:29,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:29,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:54:29,851 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:54:29,851 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:54:29,851 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:54:29,851 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:54:29,851 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:54:29,851 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:54:29,852 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:29,852 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:29,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:29,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:29,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:29,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:29,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:29,852 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:29,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:29,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:29,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:29,853 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:29,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:29,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:29,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:29,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:29,854 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:29,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:29,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:29,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:29,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:29,856 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:29,856 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:29,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:29,856 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:29,857 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:29,857 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:29,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:29,856 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:54:29,857 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:54:29,857 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:29,857 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:29,857 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:29,857 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:29,857 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:29,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:29,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:29,857 DEBUG [Thread-837] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:54:29,857 DEBUG [Thread-837] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:54:29,957 DEBUG [Thread-837] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:54:29,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:54:29,959 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:54:29,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:54:29,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:54:29,960 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:54:29,960 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:29,960 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:54:29,960 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:54:29,961 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:29,961 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:29,965 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:29,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:29,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:29,965 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:29,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:29,966 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:29,966 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:54:29,966 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:29,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:54:29,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:29,966 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:54:29,966 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:54:29,967 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:54:29,967 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:54:29,967 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:29,967 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:54:29,967 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:54:29,967 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:54:29,967 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:29,967 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. started... 2023-04-19 20:54:29,967 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. as already flushing 2023-04-19 20:54:29,967 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:29,967 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:54:29,967 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:54:29,967 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:29,967 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:54:29,967 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:54:29,969 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:29,969 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:29,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:29,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:29,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:29,969 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:29,969 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:54:29,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:29,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:29,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:29,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:29,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:29,971 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:54:29,971 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@64641219[Count = 0] remaining members to acquire global barrier 2023-04-19 20:54:29,971 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:54:29,971 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:29,972 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:29,972 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:29,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:29,972 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:54:29,972 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:54:29,972 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:29,972 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:54:29,972 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:54:29,973 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:29,973 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:54:29,973 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:29,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:29,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:29,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:29,974 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:54:29,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:29,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:29,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:29,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:29,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:29,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:29,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:29,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:54:29,976 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:54:29,976 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:54:29,976 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:54:29,976 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:54:29,976 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:54:29,976 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:54:29,982 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:29,982 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:29,982 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:29,982 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:29,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:29,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:29,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:29,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:29,982 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:29,982 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:29,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:29,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:29,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:29,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:29,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:29,983 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:29,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:29,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:29,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:29,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:29,986 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:29,986 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:29,986 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:29,986 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:29,986 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:29,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:29,986 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:29,986 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:54:29,986 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:54:29,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:29,986 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:29,987 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:29,987 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:29,987 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:29,987 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:54:29,987 DEBUG [Thread-837] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:54:29,987 DEBUG [Thread-837] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:54:29,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:29,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:30,087 DEBUG [Thread-837] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:54:30,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:54:30,089 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:54:30,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:54:30,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:54:30,093 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:54:30,093 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:30,093 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:54:30,093 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:54:30,093 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:30,093 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:30,095 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:30,095 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:30,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:30,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:30,095 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:30,095 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:54:30,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:30,095 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:30,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:54:30,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:30,096 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:54:30,096 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:54:30,096 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:54:30,096 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:54:30,096 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:30,096 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:54:30,096 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:54:30,097 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:54:30,097 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:30,097 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. started... 2023-04-19 20:54:30,097 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. as already flushing 2023-04-19 20:54:30,097 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:30,097 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:54:30,097 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:54:30,097 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:30,097 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:54:30,097 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:54:30,098 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:30,098 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:30,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:30,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:30,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:30,099 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:30,099 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:54:30,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:30,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:30,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:30,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:30,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:30,100 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:54:30,100 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@37feb7b6[Count = 0] remaining members to acquire global barrier 2023-04-19 20:54:30,100 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:54:30,100 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:30,106 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:30,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:30,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:30,106 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:54:30,106 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:54:30,106 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:30,106 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:54:30,106 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:54:30,108 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:30,108 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:54:30,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:30,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:30,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:30,108 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:30,108 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:54:30,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:30,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:30,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:30,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:30,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:30,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:30,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:30,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:54:30,110 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:54:30,110 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:54:30,110 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:54:30,111 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:54:30,111 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:54:30,111 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:54:30,113 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:30,113 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:30,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:30,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:30,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:30,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:30,113 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:30,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:30,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:30,113 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:30,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:30,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:30,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:30,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:30,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:30,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:30,115 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:30,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:30,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:30,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:30,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:30,117 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:30,117 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:30,117 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:30,117 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:30,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:30,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:30,118 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:30,118 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:30,118 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:30,118 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:30,118 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:54:30,118 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:54:30,118 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:54:30,118 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:30,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:30,118 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:30,118 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:30,118 DEBUG [Thread-837] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:54:30,118 DEBUG [Thread-837] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:54:30,133 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:30,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56352 deadline: 1681937730129, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:30,191 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:30,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56368 deadline: 1681937730191, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:30,218 DEBUG [Thread-837] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:54:30,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:54:30,221 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:54:30,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:54:30,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:54:30,222 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:54:30,222 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:30,222 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:54:30,222 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:54:30,222 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:30,222 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:30,225 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:30,225 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:30,225 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:30,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:30,225 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:30,226 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:54:30,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:30,226 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:30,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:54:30,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:30,226 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:54:30,227 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:54:30,227 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:54:30,227 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:54:30,227 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:30,227 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:54:30,227 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:54:30,227 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:54:30,227 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:30,227 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. started... 2023-04-19 20:54:30,227 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. as already flushing 2023-04-19 20:54:30,227 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:30,227 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:54:30,227 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:54:30,227 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:30,227 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:54:30,227 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:54:30,229 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:30,229 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:30,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:30,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:30,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:30,229 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:30,229 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:54:30,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:30,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:30,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:30,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:30,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:30,231 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:54:30,231 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c92e5c9[Count = 0] remaining members to acquire global barrier 2023-04-19 20:54:30,231 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:54:30,231 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:30,232 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:30,232 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:30,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:30,232 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:54:30,232 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:30,232 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:54:30,232 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:54:30,232 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:54:30,234 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:30,234 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:54:30,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:30,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:30,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:30,234 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:30,234 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:54:30,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:30,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:30,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:30,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:30,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:30,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:30,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:30,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:54:30,236 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:54:30,236 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:54:30,236 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:54:30,236 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:54:30,237 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:54:30,237 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:54:30,239 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:30,239 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:30,239 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:30,239 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:30,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:30,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:30,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:30,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:30,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:30,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:30,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:30,240 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:30,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:30,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:30,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:30,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:30,241 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:30,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:30,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:30,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:30,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:30,243 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:30,243 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:30,243 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:30,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:30,244 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:30,244 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:54:30,244 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:54:30,244 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:30,244 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:30,244 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:30,244 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:30,244 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:30,244 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:30,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:30,244 DEBUG [Thread-837] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:54:30,244 DEBUG [Thread-837] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:54:30,244 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:54:30,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:30,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:30,248 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=248 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/C/62ef60b328604a43ba048fec23bf9792 2023-04-19 20:54:30,252 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/A/1d011e0999b9447392914e99e6e1fbd3 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/1d011e0999b9447392914e99e6e1fbd3 2023-04-19 20:54:30,256 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/1d011e0999b9447392914e99e6e1fbd3, entries=400, sequenceid=248, filesize=59.4 K 2023-04-19 20:54:30,257 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/B/c3be8268e328449f998c7138ee775f6b as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/c3be8268e328449f998c7138ee775f6b 2023-04-19 20:54:30,261 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/c3be8268e328449f998c7138ee775f6b, entries=150, sequenceid=248, filesize=11.8 K 2023-04-19 20:54:30,262 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/C/62ef60b328604a43ba048fec23bf9792 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/62ef60b328604a43ba048fec23bf9792 2023-04-19 20:54:30,266 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/62ef60b328604a43ba048fec23bf9792, entries=150, sequenceid=248, filesize=11.8 K 2023-04-19 20:54:30,266 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for c7ee8087efe9c3ae02af767e46864c85 in 1286ms, sequenceid=248, compaction requested=true 2023-04-19 20:54:30,266 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c7ee8087efe9c3ae02af767e46864c85: 2023-04-19 20:54:30,266 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 20:54:30,267 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 20:54:30,267 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-19 20:54:30,267 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-19 20:54:30,267 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 20:54:30,268 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48520 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-19 20:54:30,268 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1912): c7ee8087efe9c3ae02af767e46864c85/B is initiating minor compaction (all files) 2023-04-19 20:54:30,268 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 138918 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-19 20:54:30,268 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c7ee8087efe9c3ae02af767e46864c85/B in TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:30,268 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1912): c7ee8087efe9c3ae02af767e46864c85/A is initiating minor compaction (all files) 2023-04-19 20:54:30,268 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/efcb2c3696da4e81a524a316e83b9536, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/6d3341a472ae4b2896debd09305e0c4a, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/a19cd3aeeab94e3b829aa79e558ca40f, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/c3be8268e328449f998c7138ee775f6b] into tmpdir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp, totalSize=47.4 K 2023-04-19 20:54:30,269 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c7ee8087efe9c3ae02af767e46864c85/A in TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:30,269 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/453272501ae24401b5450a3a6abca5da, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/536796bf8ff64c8aaef8f2c36c3939ca, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/e6ed8f4c953c463d894ef3e50a5c1419, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/1d011e0999b9447392914e99e6e1fbd3] into tmpdir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp, totalSize=135.7 K 2023-04-19 20:54:30,269 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting efcb2c3696da4e81a524a316e83b9536, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=196, earliestPutTs=1681937666135 2023-04-19 20:54:30,269 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 453272501ae24401b5450a3a6abca5da, keycount=150, bloomtype=ROW, size=25.7 K, encoding=NONE, compression=NONE, seqNum=196, earliestPutTs=1681937666135 2023-04-19 20:54:30,270 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 6d3341a472ae4b2896debd09305e0c4a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=221, earliestPutTs=1681937666177 2023-04-19 20:54:30,270 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 536796bf8ff64c8aaef8f2c36c3939ca, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=221, earliestPutTs=1681937666177 2023-04-19 20:54:30,270 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting a19cd3aeeab94e3b829aa79e558ca40f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=235, earliestPutTs=1681937666790 2023-04-19 20:54:30,270 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting e6ed8f4c953c463d894ef3e50a5c1419, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=235, earliestPutTs=1681937666790 2023-04-19 20:54:30,270 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting c3be8268e328449f998c7138ee775f6b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=248, earliestPutTs=1681937668977 2023-04-19 20:54:30,271 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 1d011e0999b9447392914e99e6e1fbd3, keycount=400, bloomtype=ROW, size=59.4 K, encoding=NONE, compression=NONE, seqNum=248, earliestPutTs=1681937668947 2023-04-19 20:54:30,293 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] throttle.PressureAwareThroughputController(145): c7ee8087efe9c3ae02af767e46864c85#B#compaction#150 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 20:54:30,304 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] throttle.PressureAwareThroughputController(145): c7ee8087efe9c3ae02af767e46864c85#A#compaction#151 average throughput is 1.97 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 20:54:30,317 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/B/fdc6a3f0271347078b1d3833584608a6 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/fdc6a3f0271347078b1d3833584608a6 2023-04-19 20:54:30,322 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in c7ee8087efe9c3ae02af767e46864c85/B of c7ee8087efe9c3ae02af767e46864c85 into fdc6a3f0271347078b1d3833584608a6(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 20:54:30,323 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c7ee8087efe9c3ae02af767e46864c85: 2023-04-19 20:54:30,323 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85., storeName=c7ee8087efe9c3ae02af767e46864c85/B, priority=12, startTime=1681937670267; duration=0sec 2023-04-19 20:54:30,323 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 20:54:30,323 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c7ee8087efe9c3ae02af767e46864c85 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-19 20:54:30,323 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-19 20:54:30,323 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c7ee8087efe9c3ae02af767e46864c85, store=A 2023-04-19 20:54:30,323 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 20:54:30,323 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c7ee8087efe9c3ae02af767e46864c85, store=B 2023-04-19 20:54:30,323 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 20:54:30,323 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c7ee8087efe9c3ae02af767e46864c85, store=C 2023-04-19 20:54:30,324 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 20:54:30,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(9158): Flush requested on c7ee8087efe9c3ae02af767e46864c85 2023-04-19 20:54:30,326 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48520 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-19 20:54:30,326 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1912): c7ee8087efe9c3ae02af767e46864c85/C is initiating minor compaction (all files) 2023-04-19 20:54:30,326 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c7ee8087efe9c3ae02af767e46864c85/C in TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:30,326 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/dbcb253d0c6446ac9d7b512ae1f30263, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/469dabb7d280474eadb90a7abfc38076, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/9aa7e84de7f24bd5a24e41acbc9f3709, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/62ef60b328604a43ba048fec23bf9792] into tmpdir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp, totalSize=47.4 K 2023-04-19 20:54:30,327 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting dbcb253d0c6446ac9d7b512ae1f30263, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=196, earliestPutTs=1681937666135 2023-04-19 20:54:30,327 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 469dabb7d280474eadb90a7abfc38076, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=221, earliestPutTs=1681937666177 2023-04-19 20:54:30,328 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 9aa7e84de7f24bd5a24e41acbc9f3709, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=235, earliestPutTs=1681937666790 2023-04-19 20:54:30,328 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 62ef60b328604a43ba048fec23bf9792, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=248, earliestPutTs=1681937668977 2023-04-19 20:54:30,363 DEBUG [Thread-837] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:54:30,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:54:30,365 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] throttle.PressureAwareThroughputController(145): c7ee8087efe9c3ae02af767e46864c85#C#compaction#152 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 20:54:30,366 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:54:30,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:54:30,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:54:30,368 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:54:30,368 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:30,368 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:54:30,368 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:54:30,368 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:30,368 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:30,369 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:30,369 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:30,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:30,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:30,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:30,370 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:30,370 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:54:30,370 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:30,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56326 deadline: 1681937730367, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:30,370 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:30,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:54:30,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:30,371 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:54:30,371 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:54:30,371 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:54:30,371 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:30,371 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:54:30,372 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:54:30,372 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:54:30,372 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:30,372 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. started... 2023-04-19 20:54:30,372 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. as already flushing 2023-04-19 20:54:30,372 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:30,372 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:54:30,372 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:54:30,372 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:30,372 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:54:30,372 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:54:30,374 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:30,374 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:30,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:30,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:30,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:30,375 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:30,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:30,375 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:54:30,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:30,376 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:30,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56408 deadline: 1681937730371, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:30,376 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:30,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:30,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56384 deadline: 1681937730372, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:30,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:30,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:30,377 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:54:30,378 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3310ce92[Count = 0] remaining members to acquire global barrier 2023-04-19 20:54:30,378 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:54:30,378 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:30,380 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:30,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:30,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:30,380 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:54:30,381 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:54:30,381 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:30,381 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:54:30,381 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:54:30,382 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:54:30,382 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:30,382 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:30,382 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:54:30,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:30,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:30,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:30,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:30,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:30,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:30,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:30,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:30,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:30,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:30,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:54:30,385 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:54:30,385 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:54:30,385 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:54:30,385 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:54:30,385 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:54:30,385 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:54:30,387 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:30,387 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:30,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:30,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:30,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:30,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:30,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:30,387 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:30,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:30,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:30,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:30,387 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:30,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:30,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:30,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:30,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:30,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:30,389 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:30,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:30,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:30,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:30,397 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:30,397 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:30,397 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:30,397 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:54:30,397 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:54:30,397 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:30,397 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:30,397 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:30,397 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:30,397 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:30,397 DEBUG [Thread-837] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:54:30,397 DEBUG [Thread-837] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:54:30,397 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:30,398 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:30,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:30,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:30,398 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:54:30,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:30,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:30,472 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:30,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56326 deadline: 1681937730471, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:30,481 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:30,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56408 deadline: 1681937730477, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:30,481 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:30,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56384 deadline: 1681937730477, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:30,498 DEBUG [Thread-837] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:54:30,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:54:30,499 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:54:30,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:54:30,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:54:30,501 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:54:30,501 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:30,501 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:54:30,501 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:54:30,501 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:30,501 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:30,502 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:30,502 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:30,502 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:30,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:30,503 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:30,503 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:54:30,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:30,503 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:30,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:54:30,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:30,503 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:54:30,504 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:54:30,504 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:54:30,504 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:54:30,504 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:30,504 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:54:30,504 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:54:30,504 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:54:30,504 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:30,504 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. started... 2023-04-19 20:54:30,504 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. as already flushing 2023-04-19 20:54:30,504 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:30,504 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:54:30,504 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:54:30,504 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:30,504 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:54:30,504 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:54:30,506 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:30,506 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:30,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:30,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:30,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:30,506 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:30,506 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:54:30,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:30,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:30,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:30,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:30,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:30,508 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:54:30,508 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@77a0e633[Count = 0] remaining members to acquire global barrier 2023-04-19 20:54:30,508 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:54:30,508 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:30,509 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:30,509 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:30,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:30,509 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:54:30,509 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:54:30,509 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:54:30,509 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:30,509 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:54:30,512 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:30,512 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:54:30,512 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:30,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:30,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:30,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:30,512 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:54:30,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:30,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:30,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:30,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:30,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:30,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:30,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:30,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:54:30,515 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:54:30,515 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:54:30,515 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:54:30,515 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:54:30,515 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:54:30,515 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:54:30,516 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:30,516 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:30,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:30,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:30,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:30,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:30,516 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:30,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:30,517 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:30,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:30,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:30,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:30,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:30,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:30,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:30,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:30,518 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:30,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:30,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:30,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:30,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:30,520 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:30,520 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:30,520 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:30,520 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:30,520 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:30,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:30,520 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:30,520 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:30,520 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:54:30,521 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:54:30,521 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:30,521 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:54:30,521 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:30,521 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:30,521 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:30,521 DEBUG [Thread-837] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:54:30,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:30,521 DEBUG [Thread-837] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:54:30,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:30,621 DEBUG [Thread-837] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:54:30,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:54:30,623 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:54:30,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:54:30,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:54:30,625 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:54:30,625 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:30,625 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:54:30,625 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:54:30,625 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:30,625 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:30,627 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:30,627 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:30,627 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:30,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:30,627 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:30,627 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:54:30,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:30,628 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:30,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:54:30,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:30,628 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:54:30,628 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:54:30,628 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:54:30,628 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:54:30,628 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:30,628 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:54:30,629 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:54:30,629 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:54:30,629 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:30,629 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. started... 2023-04-19 20:54:30,629 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. as already flushing 2023-04-19 20:54:30,629 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:30,629 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:54:30,629 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:54:30,629 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:30,629 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:54:30,629 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:54:30,631 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:30,631 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:30,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:30,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:30,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:30,631 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:30,631 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:54:30,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:30,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:30,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:30,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:30,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:30,633 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:54:30,633 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@35a1ac76[Count = 0] remaining members to acquire global barrier 2023-04-19 20:54:30,633 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:54:30,633 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:30,634 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:30,634 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:30,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:30,634 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:54:30,634 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:54:30,634 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:30,635 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:54:30,634 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:54:30,636 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:30,636 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:54:30,636 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:30,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:30,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:30,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:30,636 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:54:30,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:30,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:30,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:30,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:30,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:30,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:30,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:30,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:54:30,640 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:54:30,640 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:54:30,640 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:54:30,640 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:54:30,640 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:54:30,640 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:54:30,641 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:30,641 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:30,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:30,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:30,641 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:30,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:30,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:30,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:30,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:30,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:30,642 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:30,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:30,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:30,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:30,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:30,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:30,643 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:30,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:30,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:30,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:30,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:30,651 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:30,651 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:30,651 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:30,651 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:30,651 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:30,651 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:54:30,651 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:54:30,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:30,651 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:30,651 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:54:30,651 DEBUG [Thread-837] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:54:30,651 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:30,651 DEBUG [Thread-837] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:54:30,652 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:30,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:30,652 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:30,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:30,652 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:30,652 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:30,674 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:30,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56326 deadline: 1681937730673, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:30,684 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:30,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56384 deadline: 1681937730682, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:30,684 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:30,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56408 deadline: 1681937730683, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:30,752 DEBUG [Thread-837] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:54:30,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:54:30,753 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:54:30,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:54:30,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:54:30,754 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:54:30,754 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:30,754 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:54:30,754 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:54:30,755 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:30,755 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:30,756 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:30,756 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:30,756 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:30,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:30,756 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:30,756 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:54:30,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:30,757 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:30,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:54:30,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:30,757 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:54:30,757 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:54:30,757 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:54:30,757 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:54:30,757 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:30,757 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:54:30,758 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:54:30,758 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:54:30,758 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:30,758 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. started... 2023-04-19 20:54:30,758 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. as already flushing 2023-04-19 20:54:30,758 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:30,758 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:54:30,758 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:54:30,758 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:30,758 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:54:30,758 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:54:30,759 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:30,759 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:30,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:30,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:30,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:30,760 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:30,760 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:54:30,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:30,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:30,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:30,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:30,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:30,761 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:54:30,761 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ed6bba6[Count = 0] remaining members to acquire global barrier 2023-04-19 20:54:30,761 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:54:30,761 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:30,762 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:30,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:30,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:30,763 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:54:30,763 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:54:30,763 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:54:30,763 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:30,763 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:54:30,765 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:30,765 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:54:30,765 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:30,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:30,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:30,765 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:54:30,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:30,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:30,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:30,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:30,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:30,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:30,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:30,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:30,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:54:30,767 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:54:30,767 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:54:30,767 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:54:30,767 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:54:30,767 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:54:30,767 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:54:30,769 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:30,769 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:30,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:30,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:30,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:30,769 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:30,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:30,769 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:30,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:30,774 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304191c6ec0b91acd4661ada72f76bfe5c2d2 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304191c6ec0b91acd4661ada72f76bfe5c2d2 2023-04-19 20:54:30,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:30,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:30,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:30,777 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:30,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:30,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:30,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:30,779 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:30,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:30,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:30,780 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/C/14718df5e08f41daa6027bbc6b8a1339 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/14718df5e08f41daa6027bbc6b8a1339 2023-04-19 20:54:30,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:30,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:30,786 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:30,786 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:30,786 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:30,786 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:30,786 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:30,786 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:30,786 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:54:30,786 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:54:30,786 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in c7ee8087efe9c3ae02af767e46864c85/C of c7ee8087efe9c3ae02af767e46864c85 into 14718df5e08f41daa6027bbc6b8a1339(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 20:54:30,786 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c7ee8087efe9c3ae02af767e46864c85: 2023-04-19 20:54:30,786 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85., storeName=c7ee8087efe9c3ae02af767e46864c85/C, priority=12, startTime=1681937670267; duration=0sec 2023-04-19 20:54:30,786 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:30,786 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:30,787 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:30,787 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:30,787 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:30,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:30,787 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 20:54:30,786 DEBUG [Thread-837] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:54:30,787 DEBUG [Thread-837] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:54:30,787 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:30,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:30,790 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=286, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/A/44337a04a964422b9470e653acd35d82 2023-04-19 20:54:30,887 DEBUG [Thread-837] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:54:30,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:54:30,889 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:54:30,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:54:30,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:54:30,890 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:54:30,890 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:30,890 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:54:30,890 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:54:30,890 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:30,890 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:30,893 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:30,893 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:30,893 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:30,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:30,894 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:30,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:30,894 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:54:30,894 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:30,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:54:30,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:30,894 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:54:30,894 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:54:30,895 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:54:30,895 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:54:30,895 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:30,895 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:54:30,895 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:54:30,895 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:54:30,895 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:30,895 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. started... 2023-04-19 20:54:30,895 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. as already flushing 2023-04-19 20:54:30,895 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:30,895 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:54:30,895 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:54:30,895 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:30,895 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:54:30,895 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:54:30,897 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:30,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:30,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:30,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:30,897 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:30,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:30,898 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:30,898 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:54:30,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:30,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:30,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:30,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:30,899 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:54:30,899 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b079eb0[Count = 0] remaining members to acquire global barrier 2023-04-19 20:54:30,899 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:54:30,899 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:30,900 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:30,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:30,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:30,901 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:54:30,901 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:54:30,901 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:30,901 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:54:30,901 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:54:30,902 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:54:30,902 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:30,902 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:30,902 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:54:30,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:30,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:30,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:30,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:30,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:30,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:30,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:30,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:30,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:30,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:30,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:54:30,905 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:54:30,905 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:54:30,905 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:54:30,905 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:54:30,905 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:54:30,905 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:54:30,907 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:30,907 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:30,907 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:30,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:30,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:30,907 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:30,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:30,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:30,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:30,908 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:30,908 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:30,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:30,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:30,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:30,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:30,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:30,909 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:30,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:30,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:30,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:30,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:30,911 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:30,911 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:30,911 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:30,911 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:30,911 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:30,911 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:30,911 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:30,911 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:54:30,912 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:30,912 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:54:30,912 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:54:30,912 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:30,912 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:30,912 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:30,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:30,912 DEBUG [Thread-837] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:54:30,912 DEBUG [Thread-837] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:54:30,912 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:30,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:30,977 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:30,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56326 deadline: 1681937730975, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:30,987 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:30,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56408 deadline: 1681937730985, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:30,987 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:30,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56384 deadline: 1681937730986, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:31,012 DEBUG [Thread-837] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:54:31,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:54:31,014 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:54:31,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:54:31,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:54:31,016 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:54:31,016 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:31,016 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:54:31,016 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:54:31,016 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:31,016 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:31,017 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:31,017 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:31,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:31,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:31,018 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:31,018 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:54:31,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:31,018 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:31,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:54:31,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:31,019 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:54:31,019 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:54:31,019 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:54:31,019 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:54:31,019 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:31,019 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:54:31,019 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:54:31,019 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:54:31,019 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:31,019 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. started... 2023-04-19 20:54:31,019 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. as already flushing 2023-04-19 20:54:31,019 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:31,020 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:54:31,020 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:54:31,020 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:31,020 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:54:31,020 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:54:31,021 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:31,021 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:31,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:31,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:31,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:31,021 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:31,021 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:54:31,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:31,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:31,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:31,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:31,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:31,023 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:54:31,023 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3be6789d[Count = 0] remaining members to acquire global barrier 2023-04-19 20:54:31,023 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:54:31,023 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:31,029 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:31,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:31,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:31,029 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:54:31,029 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:54:31,029 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:54:31,029 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:31,029 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:54:31,030 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:31,031 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:54:31,031 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:31,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:31,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:31,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:31,031 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:54:31,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:31,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:31,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:31,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:31,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:31,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:31,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:31,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:54:31,033 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:54:31,033 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:54:31,033 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:54:31,033 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:54:31,033 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:54:31,033 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:54:31,035 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:31,035 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:31,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:31,035 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:31,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:31,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:31,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:31,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:31,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:31,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:31,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:31,035 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:31,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:31,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:31,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:31,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:31,036 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:31,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:31,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:31,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:31,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:31,038 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:31,038 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:31,038 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:31,039 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:31,039 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:31,039 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:31,039 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:31,039 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:31,039 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:54:31,039 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:54:31,039 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:31,039 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:31,039 DEBUG [Thread-837] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:54:31,039 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:31,039 DEBUG [Thread-837] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:54:31,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:31,039 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:54:31,039 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:31,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:31,139 DEBUG [Thread-837] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:54:31,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:54:31,142 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:54:31,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:54:31,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:54:31,143 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:54:31,143 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:31,143 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:54:31,143 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:54:31,144 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:31,144 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:31,144 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:31,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56352 deadline: 1681937731143, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:31,145 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/A/29a2064866c94ca38323ebf3efa79cb4 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/29a2064866c94ca38323ebf3efa79cb4 2023-04-19 20:54:31,146 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:31,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:31,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:31,147 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:31,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:31,147 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:31,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:54:31,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:31,148 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:54:31,148 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:31,148 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:54:31,148 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:54:31,148 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:54:31,148 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:54:31,148 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:31,148 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:54:31,149 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:54:31,149 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:54:31,149 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:31,149 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. started... 2023-04-19 20:54:31,149 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. as already flushing 2023-04-19 20:54:31,149 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:31,149 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:54:31,149 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:54:31,149 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:31,149 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:54:31,149 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:54:31,151 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in c7ee8087efe9c3ae02af767e46864c85/A of c7ee8087efe9c3ae02af767e46864c85 into 29a2064866c94ca38323ebf3efa79cb4(size=25.8 K), total size for store is 25.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 20:54:31,151 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c7ee8087efe9c3ae02af767e46864c85: 2023-04-19 20:54:31,151 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85., storeName=c7ee8087efe9c3ae02af767e46864c85/A, priority=12, startTime=1681937670266; duration=0sec 2023-04-19 20:54:31,151 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:31,151 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:31,151 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 20:54:31,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:31,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:31,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:31,152 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:31,152 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:54:31,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:31,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:31,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:31,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:31,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:31,153 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:54:31,153 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@31ca18b[Count = 0] remaining members to acquire global barrier 2023-04-19 20:54:31,153 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:54:31,153 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:31,154 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:31,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:31,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:31,154 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:54:31,155 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:54:31,155 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:54:31,155 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:31,155 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:54:31,156 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:31,156 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:54:31,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:31,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:31,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:31,156 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:31,156 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:54:31,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:31,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:31,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:31,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:31,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:31,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:31,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:31,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:54:31,158 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:54:31,158 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:54:31,158 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:54:31,158 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:54:31,158 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:54:31,158 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:54:31,160 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:31,160 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:31,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:31,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:31,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:31,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:31,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:31,161 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:31,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:31,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:31,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:31,161 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:31,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:31,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:31,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:31,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:31,162 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:31,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:31,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:31,164 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:31,164 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:31,164 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:31,164 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:31,164 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:31,165 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:54:31,165 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:54:31,165 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:31,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:31,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:31,164 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:31,164 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:31,165 DEBUG [Thread-837] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:54:31,165 DEBUG [Thread-837] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:54:31,165 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:31,165 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:31,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:31,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:31,171 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:54:31,201 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=286 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/B/96cda4f49f154fc58d353aa5eac8ee36 2023-04-19 20:54:31,202 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:31,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56368 deadline: 1681937731199, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:31,215 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=286 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/C/4217aa3d5153432cb855379e8c506418 2023-04-19 20:54:31,220 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/A/44337a04a964422b9470e653acd35d82 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/44337a04a964422b9470e653acd35d82 2023-04-19 20:54:31,225 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/44337a04a964422b9470e653acd35d82, entries=200, sequenceid=286, filesize=32.3 K 2023-04-19 20:54:31,226 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/B/96cda4f49f154fc58d353aa5eac8ee36 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/96cda4f49f154fc58d353aa5eac8ee36 2023-04-19 20:54:31,233 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/96cda4f49f154fc58d353aa5eac8ee36, entries=150, sequenceid=286, filesize=11.9 K 2023-04-19 20:54:31,234 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/C/4217aa3d5153432cb855379e8c506418 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/4217aa3d5153432cb855379e8c506418 2023-04-19 20:54:31,238 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/4217aa3d5153432cb855379e8c506418, entries=150, sequenceid=286, filesize=11.9 K 2023-04-19 20:54:31,239 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=53.67 KB/54960 for c7ee8087efe9c3ae02af767e46864c85 in 916ms, sequenceid=286, compaction requested=false 2023-04-19 20:54:31,239 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c7ee8087efe9c3ae02af767e46864c85: 2023-04-19 20:54:31,265 DEBUG [Thread-837] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:54:31,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:54:31,267 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:54:31,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:54:31,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:54:31,268 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:54:31,269 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:31,269 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:54:31,269 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:54:31,269 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:31,269 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:31,272 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:31,272 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:31,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:31,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:31,273 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:31,273 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:54:31,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:31,273 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:31,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:54:31,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:31,274 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:54:31,274 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:54:31,274 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:54:31,274 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:54:31,274 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:31,274 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:54:31,274 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:54:31,274 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:54:31,274 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:31,274 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. started... 2023-04-19 20:54:31,274 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c7ee8087efe9c3ae02af767e46864c85 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-19 20:54:31,275 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c7ee8087efe9c3ae02af767e46864c85, store=A 2023-04-19 20:54:31,275 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:54:31,275 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c7ee8087efe9c3ae02af767e46864c85, store=B 2023-04-19 20:54:31,275 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:54:31,275 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c7ee8087efe9c3ae02af767e46864c85, store=C 2023-04-19 20:54:31,275 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:54:31,306 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304193a9c6352d363476283bc46dfcd3549c8 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304193a9c6352d363476283bc46dfcd3549c8 2023-04-19 20:54:31,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(9158): Flush requested on c7ee8087efe9c3ae02af767e46864c85 2023-04-19 20:54:31,483 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. as already flushing 2023-04-19 20:54:31,508 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:31,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56326 deadline: 1681937731505, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:31,509 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681937671501, max timestamp=1681937671507Num uniques 859; 2023-04-19 20:54:31,509 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-19 20:54:31,509 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681937671501, max timestamp=1681937671507Num uniques 859; 2023-04-19 20:54:31,509 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-19 20:54:31,510 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681937671501, max timestamp=1681937671507Num uniques 859; 2023-04-19 20:54:31,510 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-19 20:54:31,533 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:31,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56384 deadline: 1681937731532, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:31,537 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:31,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56408 deadline: 1681937731534, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:31,610 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:31,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56326 deadline: 1681937731610, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:31,636 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:31,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56384 deadline: 1681937731634, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:31,639 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:31,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56408 deadline: 1681937731638, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:31,712 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=299, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/A/ecf546bea22e433fbaf955afcea8a3cf 2023-04-19 20:54:31,723 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=299 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/B/3fd5c0de13ed46c6ad526a162460340c 2023-04-19 20:54:31,733 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=299 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/C/d1f1b9052223440fa3ccec51bd74a9f8 2023-04-19 20:54:31,737 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/A/ecf546bea22e433fbaf955afcea8a3cf as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/ecf546bea22e433fbaf955afcea8a3cf 2023-04-19 20:54:31,742 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/ecf546bea22e433fbaf955afcea8a3cf, entries=150, sequenceid=299, filesize=25.4 K 2023-04-19 20:54:31,742 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/B/3fd5c0de13ed46c6ad526a162460340c as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/3fd5c0de13ed46c6ad526a162460340c 2023-04-19 20:54:31,746 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/3fd5c0de13ed46c6ad526a162460340c, entries=150, sequenceid=299, filesize=11.9 K 2023-04-19 20:54:31,747 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/C/d1f1b9052223440fa3ccec51bd74a9f8 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/d1f1b9052223440fa3ccec51bd74a9f8 2023-04-19 20:54:31,753 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/d1f1b9052223440fa3ccec51bd74a9f8, entries=150, sequenceid=299, filesize=11.9 K 2023-04-19 20:54:31,754 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for c7ee8087efe9c3ae02af767e46864c85 in 480ms, sequenceid=299, compaction requested=true 2023-04-19 20:54:31,754 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c7ee8087efe9c3ae02af767e46864c85: 2023-04-19 20:54:31,754 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:31,754 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:54:31,754 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:54:31,754 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:31,754 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:54:31,754 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:54:31,759 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:31,759 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:31,759 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:54:31,760 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:31,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:31,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:31,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:31,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:31,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:31,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:31,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:31,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:31,762 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:54:31,762 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@485c21c5[Count = 0] remaining members to acquire global barrier 2023-04-19 20:54:31,762 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:54:31,762 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:31,768 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:31,768 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:31,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:31,768 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:54:31,768 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:54:31,768 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:54:31,768 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:31,768 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:54:31,770 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:54:31,770 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:31,770 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:31,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:31,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:31,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:31,770 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:54:31,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:31,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:31,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:31,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:31,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:31,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:31,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:31,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:54:31,773 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:54:31,773 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:54:31,773 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:54:31,773 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:54:31,773 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:54:31,773 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:54:31,775 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:31,775 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:31,775 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:31,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:31,775 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:31,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:31,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:31,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:31,775 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:31,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:31,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:31,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:31,777 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:31,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:31,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:31,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:31,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:31,780 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:31,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:31,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:31,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:31,784 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:31,784 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:31,784 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:31,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:31,785 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:31,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:31,785 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:31,785 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:31,785 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:31,785 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:31,785 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:31,785 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:31,785 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:31,785 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:31,785 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:54:31,785 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:54:31,786 DEBUG [Thread-837] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:54:31,786 DEBUG [Thread-837] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:54:31,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(9158): Flush requested on c7ee8087efe9c3ae02af767e46864c85 2023-04-19 20:54:31,814 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c7ee8087efe9c3ae02af767e46864c85 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-04-19 20:54:31,814 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c7ee8087efe9c3ae02af767e46864c85, store=A 2023-04-19 20:54:31,815 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 20:54:31,815 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c7ee8087efe9c3ae02af767e46864c85, store=B 2023-04-19 20:54:31,815 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 20:54:31,815 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c7ee8087efe9c3ae02af767e46864c85, store=C 2023-04-19 20:54:31,815 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 20:54:31,840 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230419ea2b1b564b54469abc8ce6fd48077134 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419ea2b1b564b54469abc8ce6fd48077134 2023-04-19 20:54:31,842 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:31,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56384 deadline: 1681937731839, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:31,843 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:31,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56326 deadline: 1681937731841, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:31,843 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:31,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56408 deadline: 1681937731841, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:31,886 DEBUG [Thread-837] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:54:31,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:54:31,895 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:54:31,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:54:31,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:54:31,896 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:54:31,896 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:31,897 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:54:31,897 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:54:31,897 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:31,897 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:31,899 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:31,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:31,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:31,899 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:31,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:31,900 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:31,900 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:31,900 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:54:31,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:54:31,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:31,900 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:54:31,900 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:54:31,900 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:54:31,900 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:54:31,900 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:31,900 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:54:31,902 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:54:31,902 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:54:31,902 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:31,902 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. started... 2023-04-19 20:54:31,902 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. as already flushing 2023-04-19 20:54:31,902 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:31,902 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:54:31,902 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:54:31,902 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:31,902 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:54:31,902 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:54:31,904 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:31,904 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:31,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:31,904 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:31,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:31,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:31,904 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:54:31,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:31,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:31,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:31,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:31,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:31,906 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:54:31,906 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:54:31,906 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3fd9c460[Count = 0] remaining members to acquire global barrier 2023-04-19 20:54:31,906 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:31,907 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:31,907 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:31,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:31,908 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:31,908 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:54:31,908 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:54:31,908 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:54:31,908 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:54:31,911 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:31,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:31,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:31,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:31,911 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:54:31,911 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:31,911 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:54:31,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:31,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:31,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:31,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:31,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:31,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:31,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:31,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:54:31,915 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:54:31,915 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:54:31,915 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:54:31,915 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:54:31,915 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:54:31,915 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:54:31,917 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:31,917 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:31,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:31,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:31,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:31,917 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:31,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:31,917 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:31,918 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:31,918 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:31,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:31,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:31,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:31,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:31,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:31,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:31,919 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:31,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:31,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:31,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:31,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:31,921 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:31,921 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:31,921 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:31,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:31,921 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:31,921 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:31,921 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:31,921 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:31,921 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:31,921 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:31,921 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:31,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:31,921 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:54:31,922 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:54:31,922 DEBUG [Thread-837] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:54:31,922 DEBUG [Thread-837] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:54:31,922 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:31,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:31,922 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:54:31,946 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:31,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56326 deadline: 1681937731944, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:32,022 DEBUG [Thread-837] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:54:32,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:54:32,024 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:54:32,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:54:32,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:54:32,026 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:54:32,026 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:32,026 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:54:32,026 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:54:32,026 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:32,026 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:32,027 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:32,027 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:32,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:32,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:32,028 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:32,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:32,028 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:54:32,028 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:32,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:54:32,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:32,029 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:54:32,029 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:54:32,029 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:54:32,029 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:54:32,029 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:32,029 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:54:32,029 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:54:32,029 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:54:32,029 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:32,029 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. started... 2023-04-19 20:54:32,029 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. as already flushing 2023-04-19 20:54:32,029 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:32,029 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:54:32,029 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:54:32,029 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:32,029 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:54:32,029 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:54:32,032 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:32,032 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:32,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:32,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:32,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:32,033 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:32,033 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:54:32,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:32,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:32,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:32,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:32,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:32,034 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:54:32,034 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@37f37196[Count = 0] remaining members to acquire global barrier 2023-04-19 20:54:32,034 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:54:32,034 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:32,036 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:32,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:32,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:32,036 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:54:32,036 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:54:32,036 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:54:32,037 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:32,037 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:54:32,038 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:32,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:32,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:32,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:32,038 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:54:32,038 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:32,038 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:54:32,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:32,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:32,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:32,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:32,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:32,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:32,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:32,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:54:32,040 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:54:32,040 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:54:32,040 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:54:32,040 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:54:32,041 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:54:32,041 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:54:32,042 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:32,042 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:32,042 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:32,042 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:32,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:32,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:32,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:32,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:32,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:32,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:32,043 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:32,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:32,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:32,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:32,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:32,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:32,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:32,045 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:32,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:32,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:32,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:32,048 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:32,048 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:54:32,048 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:32,048 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:32,049 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:32,048 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:54:32,048 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:32,048 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:32,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:32,049 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:32,049 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:32,049 DEBUG [Thread-837] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:54:32,049 DEBUG [Thread-837] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:54:32,049 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:32,049 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:32,049 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:32,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:32,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:32,147 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:32,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56408 deadline: 1681937732146, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:32,148 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:32,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56384 deadline: 1681937732146, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:32,149 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:32,149 DEBUG [Thread-837] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:54:32,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56326 deadline: 1681937732149, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:32,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:54:32,151 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:54:32,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:54:32,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:54:32,153 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:54:32,153 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:32,153 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:54:32,153 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:54:32,154 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:32,154 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:32,156 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:32,156 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:32,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:32,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:32,156 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:32,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:32,156 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:54:32,157 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:32,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:54:32,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:32,157 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:54:32,157 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:54:32,157 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:54:32,157 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:54:32,157 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:32,157 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:54:32,158 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:54:32,158 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:54:32,158 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:32,158 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. started... 2023-04-19 20:54:32,158 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. as already flushing 2023-04-19 20:54:32,158 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:32,158 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:54:32,158 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:54:32,158 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:32,158 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:54:32,158 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:54:32,160 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:32,160 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:32,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:32,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:32,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:32,161 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:32,161 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:54:32,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:32,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:32,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:32,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:32,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:32,162 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:54:32,162 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@51059e1a[Count = 0] remaining members to acquire global barrier 2023-04-19 20:54:32,162 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:54:32,163 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:32,167 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:32,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:32,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:32,167 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:54:32,167 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:54:32,167 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:54:32,168 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:32,168 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:54:32,169 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:32,169 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:54:32,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:32,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:32,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:32,169 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:32,169 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:54:32,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:32,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:32,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:32,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:32,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:32,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:32,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:32,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:54:32,172 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:54:32,172 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:54:32,173 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:54:32,173 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:54:32,173 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:54:32,173 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:54:32,174 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:32,174 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:32,174 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:32,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:32,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:32,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:32,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:32,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:32,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:32,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:32,175 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:32,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:32,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:32,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:32,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:32,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:32,177 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:32,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:32,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:32,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:32,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:32,179 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:32,179 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:32,179 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:32,179 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:32,179 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:32,179 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:32,179 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:32,179 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:54:32,180 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:54:32,179 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:32,180 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:54:32,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:32,180 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:32,180 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:32,180 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:32,180 DEBUG [Thread-837] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:54:32,180 DEBUG [Thread-837] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:54:32,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:32,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:32,250 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=335, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/A/70b3efd12fef43cfb355c5ecf42d90d8 2023-04-19 20:54:32,280 DEBUG [Thread-837] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:54:32,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:54:32,282 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:54:32,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:54:32,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:54:32,283 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:54:32,283 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:32,283 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:54:32,283 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:54:32,283 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:32,284 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:32,286 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:32,286 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:32,286 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:32,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:32,286 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:32,286 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:54:32,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:32,287 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:32,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:54:32,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:32,288 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:54:32,288 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:54:32,288 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:54:32,288 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:54:32,288 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:32,288 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:54:32,288 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:54:32,288 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:54:32,288 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:32,288 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. started... 2023-04-19 20:54:32,289 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. as already flushing 2023-04-19 20:54:32,289 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:32,289 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:54:32,289 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:54:32,289 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:32,289 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:54:32,289 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:54:32,290 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:32,290 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:32,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:32,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:32,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:32,290 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:32,291 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:54:32,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:32,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:32,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:32,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:32,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:32,292 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:54:32,292 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@78fedff6[Count = 0] remaining members to acquire global barrier 2023-04-19 20:54:32,292 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:54:32,292 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:32,293 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:32,293 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:32,293 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:54:32,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:32,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:32,294 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:54:32,294 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:54:32,294 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:54:32,295 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:32,295 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:54:32,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:32,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:32,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:32,295 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:32,296 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:54:32,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:32,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:32,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:32,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:32,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:32,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:32,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:32,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:54:32,298 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:54:32,298 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:54:32,298 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:54:32,298 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:54:32,298 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:54:32,298 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:54:32,300 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:32,300 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:32,300 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:32,300 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:32,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:32,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:32,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:32,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:32,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:32,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:32,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:32,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:32,301 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:32,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:32,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:32,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:32,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:32,304 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:32,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:32,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:32,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:32,307 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:32,307 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:32,307 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:32,307 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:32,307 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:32,307 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:32,307 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:32,307 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:32,307 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:54:32,307 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:32,307 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:54:32,307 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:54:32,307 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:32,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:32,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:32,307 DEBUG [Thread-837] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:54:32,307 DEBUG [Thread-837] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:54:32,308 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:32,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:32,408 DEBUG [Thread-837] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:54:32,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:54:32,410 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:54:32,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:54:32,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:54:32,411 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:54:32,411 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:32,411 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:54:32,411 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:54:32,411 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:32,411 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:32,413 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:32,413 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:32,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:32,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:32,413 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:32,413 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:54:32,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:32,413 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:32,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:54:32,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:32,414 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:54:32,414 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:54:32,414 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:54:32,414 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:54:32,414 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:32,414 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:54:32,414 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:54:32,415 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:54:32,415 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:32,415 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. started... 2023-04-19 20:54:32,415 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. as already flushing 2023-04-19 20:54:32,415 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:32,415 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:54:32,415 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:54:32,415 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:32,415 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:54:32,415 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:54:32,416 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:32,416 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:32,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:32,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:32,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:32,417 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:32,417 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:54:32,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:32,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:32,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:32,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:32,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:32,418 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:54:32,418 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@9f5ee4d[Count = 0] remaining members to acquire global barrier 2023-04-19 20:54:32,418 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:54:32,418 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:32,420 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:32,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:32,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:32,420 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:54:32,420 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:54:32,420 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:54:32,420 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:32,421 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:54:32,423 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:54:32,423 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:32,423 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:32,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:32,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:32,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:32,423 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:54:32,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:32,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:32,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:32,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:32,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:32,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:32,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:32,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:54:32,426 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:54:32,426 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:54:32,426 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:54:32,426 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:54:32,426 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:54:32,426 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:54:32,427 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:32,427 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:32,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:32,428 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:32,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:32,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:32,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:32,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:32,428 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:32,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:32,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:32,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:32,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:32,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:32,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:32,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:32,430 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:32,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:32,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:32,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:32,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:32,433 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:32,433 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:32,433 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:32,433 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:32,433 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:32,433 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:32,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:32,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:32,433 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:54:32,433 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:54:32,433 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:32,433 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:32,433 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:32,433 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:32,433 DEBUG [Thread-837] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:54:32,433 DEBUG [Thread-837] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:54:32,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:32,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:32,433 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:54:32,452 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:32,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56326 deadline: 1681937732450, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:32,533 DEBUG [Thread-837] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:54:32,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:54:32,535 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:54:32,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:54:32,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:54:32,536 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:54:32,537 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:32,537 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:54:32,537 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:54:32,537 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:32,537 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:32,539 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:32,539 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:32,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:32,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:32,539 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:32,539 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:54:32,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:32,540 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:32,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:54:32,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:32,540 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:54:32,541 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:54:32,541 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:54:32,541 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:54:32,541 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:32,541 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:54:32,541 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:54:32,541 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:54:32,541 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:32,541 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. started... 2023-04-19 20:54:32,541 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. as already flushing 2023-04-19 20:54:32,541 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:32,541 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:54:32,541 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:54:32,541 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:32,541 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:54:32,541 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:54:32,543 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:32,543 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:32,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:32,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:32,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:32,543 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:32,543 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:54:32,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:32,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:32,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:32,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:32,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:32,544 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:54:32,545 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@25264124[Count = 0] remaining members to acquire global barrier 2023-04-19 20:54:32,545 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:54:32,545 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:32,546 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:32,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:32,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:32,546 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:54:32,546 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:54:32,546 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:54:32,546 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:32,546 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:54:32,551 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:32,551 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:54:32,551 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:32,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:32,551 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:54:32,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:32,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:32,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:32,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:32,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:32,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:32,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:32,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:32,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:32,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:54:32,554 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:54:32,554 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:54:32,554 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:54:32,554 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:54:32,554 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:54:32,554 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:54:32,556 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:32,556 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:32,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:32,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:32,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:32,556 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:32,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:32,556 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:32,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:32,556 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:32,556 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:32,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:32,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:32,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:32,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:32,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:32,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:32,558 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:32,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:32,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:32,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:32,561 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:32,561 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:32,561 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:32,561 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:54:32,561 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:54:32,561 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:32,561 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:32,561 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:32,561 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:32,561 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:32,561 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:32,561 DEBUG [Thread-837] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:54:32,561 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:32,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:32,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:32,561 DEBUG [Thread-837] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:54:32,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:32,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:32,653 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:32,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56384 deadline: 1681937732650, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:32,653 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:32,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56408 deadline: 1681937732651, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:32,661 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=335 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/B/6db9953b384047febc10f2289ae5ff13 2023-04-19 20:54:32,661 DEBUG [Thread-837] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:54:32,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:54:32,664 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:54:32,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:54:32,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:54:32,666 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:54:32,666 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:32,666 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:54:32,666 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:54:32,666 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:32,666 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:32,668 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:32,668 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:32,668 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:32,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:32,668 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:32,668 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:54:32,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:32,670 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:32,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:54:32,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:32,670 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:54:32,670 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:54:32,670 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:54:32,670 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:54:32,670 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:32,670 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:54:32,671 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:54:32,671 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:54:32,671 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:32,671 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. started... 2023-04-19 20:54:32,671 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. as already flushing 2023-04-19 20:54:32,671 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:32,671 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:54:32,671 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:54:32,671 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:32,671 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:54:32,671 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:54:32,673 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:32,673 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:32,673 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=335 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/C/3250fa01f6fd4038bf1f73b7e7e4f2cb 2023-04-19 20:54:32,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:32,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:32,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:32,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:32,674 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:32,674 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:54:32,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:32,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:32,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:32,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:32,676 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:54:32,676 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@582efed4[Count = 0] remaining members to acquire global barrier 2023-04-19 20:54:32,676 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:54:32,676 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:32,679 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:32,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:32,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:32,679 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:54:32,679 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:54:32,679 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:54:32,679 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:32,679 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:54:32,681 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:32,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:32,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:32,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:32,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:32,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:32,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:32,681 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:54:32,682 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/A/70b3efd12fef43cfb355c5ecf42d90d8 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/70b3efd12fef43cfb355c5ecf42d90d8 2023-04-19 20:54:32,682 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:32,683 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:54:32,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:32,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:32,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:32,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:32,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:54:32,687 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:54:32,687 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:54:32,687 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:54:32,687 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:54:32,687 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:54:32,687 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:54:32,688 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/70b3efd12fef43cfb355c5ecf42d90d8, entries=200, sequenceid=335, filesize=32.3 K 2023-04-19 20:54:32,689 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/B/6db9953b384047febc10f2289ae5ff13 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/6db9953b384047febc10f2289ae5ff13 2023-04-19 20:54:32,693 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:32,693 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:32,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:32,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:32,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:32,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:32,693 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:32,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:32,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:32,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:32,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:32,694 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:32,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:32,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:32,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:32,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:32,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:32,695 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:32,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:32,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:32,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:32,697 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:32,697 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:32,697 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:32,697 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:32,697 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:32,697 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:32,697 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:32,697 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:54:32,698 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:32,697 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:54:32,698 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:54:32,697 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:32,698 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:32,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:32,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:32,698 DEBUG [Thread-837] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:54:32,698 DEBUG [Thread-837] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:54:32,698 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/6db9953b384047febc10f2289ae5ff13, entries=150, sequenceid=335, filesize=11.9 K 2023-04-19 20:54:32,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:32,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:32,707 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/C/3250fa01f6fd4038bf1f73b7e7e4f2cb as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/3250fa01f6fd4038bf1f73b7e7e4f2cb 2023-04-19 20:54:32,713 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/3250fa01f6fd4038bf1f73b7e7e4f2cb, entries=150, sequenceid=335, filesize=11.9 K 2023-04-19 20:54:32,713 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=46.96 KB/48090 for c7ee8087efe9c3ae02af767e46864c85 in 899ms, sequenceid=335, compaction requested=true 2023-04-19 20:54:32,714 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c7ee8087efe9c3ae02af767e46864c85: 2023-04-19 20:54:32,714 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 20:54:32,714 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-19 20:54:32,714 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-19 20:54:32,714 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 20:54:32,714 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 20:54:32,716 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49106 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-19 20:54:32,717 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1912): c7ee8087efe9c3ae02af767e46864c85/B is initiating minor compaction (all files) 2023-04-19 20:54:32,717 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c7ee8087efe9c3ae02af767e46864c85/B in TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:32,717 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/fdc6a3f0271347078b1d3833584608a6, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/96cda4f49f154fc58d353aa5eac8ee36, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/3fd5c0de13ed46c6ad526a162460340c, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/6db9953b384047febc10f2289ae5ff13] into tmpdir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp, totalSize=48.0 K 2023-04-19 20:54:32,717 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 118626 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-19 20:54:32,717 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1912): c7ee8087efe9c3ae02af767e46864c85/A is initiating minor compaction (all files) 2023-04-19 20:54:32,717 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c7ee8087efe9c3ae02af767e46864c85/A in TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:32,718 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/29a2064866c94ca38323ebf3efa79cb4, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/44337a04a964422b9470e653acd35d82, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/ecf546bea22e433fbaf955afcea8a3cf, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/70b3efd12fef43cfb355c5ecf42d90d8] into tmpdir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp, totalSize=115.8 K 2023-04-19 20:54:32,718 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting fdc6a3f0271347078b1d3833584608a6, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=248, earliestPutTs=1681937668977 2023-04-19 20:54:32,718 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 29a2064866c94ca38323ebf3efa79cb4, keycount=150, bloomtype=ROW, size=25.8 K, encoding=NONE, compression=NONE, seqNum=248, earliestPutTs=1681937668977 2023-04-19 20:54:32,719 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 44337a04a964422b9470e653acd35d82, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=286, earliestPutTs=1681937669065 2023-04-19 20:54:32,719 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 96cda4f49f154fc58d353aa5eac8ee36, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=286, earliestPutTs=1681937669065 2023-04-19 20:54:32,720 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 3fd5c0de13ed46c6ad526a162460340c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=299, earliestPutTs=1681937670326 2023-04-19 20:54:32,720 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting ecf546bea22e433fbaf955afcea8a3cf, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=299, earliestPutTs=1681937670326 2023-04-19 20:54:32,720 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 6db9953b384047febc10f2289ae5ff13, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=335, earliestPutTs=1681937671523 2023-04-19 20:54:32,721 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 70b3efd12fef43cfb355c5ecf42d90d8, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=335, earliestPutTs=1681937671523 2023-04-19 20:54:32,734 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] throttle.PressureAwareThroughputController(145): c7ee8087efe9c3ae02af767e46864c85#B#compaction#162 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 20:54:32,749 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] throttle.PressureAwareThroughputController(145): c7ee8087efe9c3ae02af767e46864c85#A#compaction#163 average throughput is 2.46 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 20:54:32,798 DEBUG [Thread-837] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:54:32,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:54:32,800 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:54:32,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:54:32,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:54:32,801 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:54:32,801 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:32,802 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:54:32,802 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:54:32,802 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:32,802 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:32,803 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:32,803 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:32,803 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:32,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:32,804 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:32,804 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:54:32,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:32,804 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:32,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:54:32,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:32,805 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:54:32,805 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:54:32,805 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:54:32,805 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:54:32,805 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:32,805 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:54:32,806 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:54:32,806 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:54:32,806 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:32,806 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. started... 2023-04-19 20:54:32,806 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c7ee8087efe9c3ae02af767e46864c85 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-19 20:54:32,806 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c7ee8087efe9c3ae02af767e46864c85, store=A 2023-04-19 20:54:32,806 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:54:32,806 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c7ee8087efe9c3ae02af767e46864c85, store=B 2023-04-19 20:54:32,806 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:54:32,807 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c7ee8087efe9c3ae02af767e46864c85, store=C 2023-04-19 20:54:32,807 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:54:32,845 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230419362b4d4299874edab5306ffb9454ca1b to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419362b4d4299874edab5306ffb9454ca1b 2023-04-19 20:54:32,858 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=345, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/A/2777891c51d64a84a3e54bfb01fc4a0f 2023-04-19 20:54:32,871 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=345 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/B/c8caf13fe1594064bf33ee22afa5d125 2023-04-19 20:54:32,891 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=345 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/C/954c3406cbda45a4b016a86d6f1c7b8c 2023-04-19 20:54:32,897 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/A/2777891c51d64a84a3e54bfb01fc4a0f as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/2777891c51d64a84a3e54bfb01fc4a0f 2023-04-19 20:54:32,902 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/2777891c51d64a84a3e54bfb01fc4a0f, entries=150, sequenceid=345, filesize=25.4 K 2023-04-19 20:54:32,904 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/B/c8caf13fe1594064bf33ee22afa5d125 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/c8caf13fe1594064bf33ee22afa5d125 2023-04-19 20:54:32,908 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/c8caf13fe1594064bf33ee22afa5d125, entries=150, sequenceid=345, filesize=11.9 K 2023-04-19 20:54:32,909 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/C/954c3406cbda45a4b016a86d6f1c7b8c as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/954c3406cbda45a4b016a86d6f1c7b8c 2023-04-19 20:54:32,914 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/954c3406cbda45a4b016a86d6f1c7b8c, entries=150, sequenceid=345, filesize=11.9 K 2023-04-19 20:54:32,915 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=0 B/0 for c7ee8087efe9c3ae02af767e46864c85 in 109ms, sequenceid=345, compaction requested=true 2023-04-19 20:54:32,915 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c7ee8087efe9c3ae02af767e46864c85: 2023-04-19 20:54:32,915 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:32,915 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:54:32,915 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:54:32,915 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:32,916 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:54:32,916 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:54:32,918 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:32,918 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:32,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:32,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:32,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:32,919 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:32,919 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:54:32,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:32,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:32,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:32,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:32,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:32,921 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:54:32,921 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@56a18b48[Count = 0] remaining members to acquire global barrier 2023-04-19 20:54:32,921 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:54:32,921 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:32,922 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:32,922 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:32,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:32,922 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:54:32,922 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:54:32,922 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:54:32,923 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:32,923 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:54:32,924 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:54:32,924 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:32,924 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:54:32,924 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:32,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:32,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:32,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:32,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:32,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:32,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:32,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:32,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:32,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:32,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:32,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:54:32,927 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:54:32,927 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:54:32,927 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:54:32,927 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:54:32,927 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:54:32,927 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:54:32,929 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:32,929 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:32,929 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:32,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:32,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:32,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:32,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:32,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:32,930 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:32,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:32,930 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:32,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:32,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:32,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:32,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:32,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:32,931 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:32,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:32,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:32,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:32,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:32,939 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:32,939 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:32,939 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:32,939 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:32,940 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:54:32,940 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:54:32,939 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:32,940 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:32,940 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:32,940 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:32,940 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:32,940 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:32,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:32,940 DEBUG [Thread-837] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:54:32,940 DEBUG [Thread-837] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:54:32,940 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:54:32,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:32,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:32,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:33,040 DEBUG [Thread-837] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:54:33,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:54:33,043 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:54:33,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:54:33,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:54:33,045 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:54:33,045 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:33,045 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:54:33,045 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:54:33,046 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:33,046 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:33,077 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:33,077 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:33,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:33,077 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:33,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:33,078 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:33,078 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:54:33,078 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:33,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:54:33,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:33,079 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:54:33,079 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:54:33,079 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:54:33,079 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:33,080 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:54:33,080 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:54:33,080 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:54:33,080 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:33,080 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. started... 2023-04-19 20:54:33,080 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c7ee8087efe9c3ae02af767e46864c85 3/3 column families, dataSize=20.13 KB heapSize=60.52 KB 2023-04-19 20:54:33,080 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c7ee8087efe9c3ae02af767e46864c85, store=A 2023-04-19 20:54:33,081 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:54:33,081 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c7ee8087efe9c3ae02af767e46864c85, store=B 2023-04-19 20:54:33,081 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:54:33,081 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c7ee8087efe9c3ae02af767e46864c85, store=C 2023-04-19 20:54:33,081 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:54:33,110 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230419e70b71edc0d64773902ac5340d8231d4 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419e70b71edc0d64773902ac5340d8231d4 2023-04-19 20:54:33,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(9158): Flush requested on c7ee8087efe9c3ae02af767e46864c85 2023-04-19 20:54:33,113 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. as already flushing 2023-04-19 20:54:33,130 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=351, memsize=6.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/A/f9fa9e3da88c43a6995013c0c70482ed 2023-04-19 20:54:33,156 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/B/8f53f8ac6c6c4962a8475ec89300ef67 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/8f53f8ac6c6c4962a8475ec89300ef67 2023-04-19 20:54:33,163 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in c7ee8087efe9c3ae02af767e46864c85/B of c7ee8087efe9c3ae02af767e46864c85 into 8f53f8ac6c6c4962a8475ec89300ef67(size=12.5 K), total size for store is 24.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 20:54:33,163 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c7ee8087efe9c3ae02af767e46864c85: 2023-04-19 20:54:33,163 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85., storeName=c7ee8087efe9c3ae02af767e46864c85/B, priority=12, startTime=1681937672714; duration=0sec 2023-04-19 20:54:33,163 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 20:54:33,164 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-19 20:54:33,164 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=351 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/B/8aa9018bc5b04eab9765f5b3b73b5244 2023-04-19 20:54:33,170 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61292 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-19 20:54:33,170 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1912): c7ee8087efe9c3ae02af767e46864c85/C is initiating minor compaction (all files) 2023-04-19 20:54:33,170 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c7ee8087efe9c3ae02af767e46864c85/C in TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:33,171 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/14718df5e08f41daa6027bbc6b8a1339, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/4217aa3d5153432cb855379e8c506418, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/d1f1b9052223440fa3ccec51bd74a9f8, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/3250fa01f6fd4038bf1f73b7e7e4f2cb, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/954c3406cbda45a4b016a86d6f1c7b8c] into tmpdir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp, totalSize=59.9 K 2023-04-19 20:54:33,171 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 14718df5e08f41daa6027bbc6b8a1339, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=248, earliestPutTs=1681937668977 2023-04-19 20:54:33,172 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 4217aa3d5153432cb855379e8c506418, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=286, earliestPutTs=1681937669065 2023-04-19 20:54:33,172 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting d1f1b9052223440fa3ccec51bd74a9f8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=299, earliestPutTs=1681937670326 2023-04-19 20:54:33,173 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 3250fa01f6fd4038bf1f73b7e7e4f2cb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=335, earliestPutTs=1681937671523 2023-04-19 20:54:33,173 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 954c3406cbda45a4b016a86d6f1c7b8c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=345, earliestPutTs=1681937671826 2023-04-19 20:54:33,177 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681937673151, max timestamp=1681937673174Num uniques 859; 2023-04-19 20:54:33,177 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-19 20:54:33,178 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681937673151, max timestamp=1681937673174Num uniques 859; 2023-04-19 20:54:33,178 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-19 20:54:33,180 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681937673151, max timestamp=1681937673174Num uniques 859; 2023-04-19 20:54:33,180 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-19 20:54:33,200 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=351 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/C/873cb6ed3c2245e7a75fc47062026599 2023-04-19 20:54:33,202 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] throttle.PressureAwareThroughputController(145): c7ee8087efe9c3ae02af767e46864c85#C#compaction#170 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 20:54:33,206 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/A/f9fa9e3da88c43a6995013c0c70482ed as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/f9fa9e3da88c43a6995013c0c70482ed 2023-04-19 20:54:33,211 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/f9fa9e3da88c43a6995013c0c70482ed, entries=150, sequenceid=351, filesize=25.4 K 2023-04-19 20:54:33,212 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/B/8aa9018bc5b04eab9765f5b3b73b5244 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/8aa9018bc5b04eab9765f5b3b73b5244 2023-04-19 20:54:33,216 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/8aa9018bc5b04eab9765f5b3b73b5244, entries=100, sequenceid=351, filesize=9.5 K 2023-04-19 20:54:33,217 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/C/873cb6ed3c2245e7a75fc47062026599 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/873cb6ed3c2245e7a75fc47062026599 2023-04-19 20:54:33,227 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1681937673197, max timestamp=1681937673224Num uniques 1068; 2023-04-19 20:54:33,227 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=12268, suffix heap size=139040,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=268 2023-04-19 20:54:33,227 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1681937673197, max timestamp=1681937673224Num uniques 1068; 2023-04-19 20:54:33,227 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=12268, suffix heap size=139040,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=268 2023-04-19 20:54:33,229 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1681937673197, max timestamp=1681937673224Num uniques 1068; 2023-04-19 20:54:33,230 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=9978, suffix heap size=139040,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=218 2023-04-19 20:54:33,231 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/873cb6ed3c2245e7a75fc47062026599, entries=100, sequenceid=351, filesize=9.5 K 2023-04-19 20:54:33,232 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.13 KB/20610, heapSize ~60.47 KB/61920, currentSize=58.14 KB/59540 for c7ee8087efe9c3ae02af767e46864c85 in 152ms, sequenceid=351, compaction requested=true 2023-04-19 20:54:33,232 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c7ee8087efe9c3ae02af767e46864c85: 2023-04-19 20:54:33,232 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:33,233 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:54:33,233 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:54:33,233 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:33,233 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:54:33,233 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:54:33,234 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:33,234 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:33,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:33,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:33,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:33,235 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:33,235 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:54:33,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:33,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:33,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:33,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(9158): Flush requested on c7ee8087efe9c3ae02af767e46864c85 2023-04-19 20:54:33,236 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c7ee8087efe9c3ae02af767e46864c85 3/3 column families, dataSize=64.85 KB heapSize=170.47 KB 2023-04-19 20:54:33,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:33,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:33,237 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:54:33,237 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@70008057[Count = 0] remaining members to acquire global barrier 2023-04-19 20:54:33,237 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:54:33,237 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:33,238 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c7ee8087efe9c3ae02af767e46864c85, store=A 2023-04-19 20:54:33,238 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 20:54:33,238 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c7ee8087efe9c3ae02af767e46864c85, store=B 2023-04-19 20:54:33,238 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 20:54:33,238 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c7ee8087efe9c3ae02af767e46864c85, store=C 2023-04-19 20:54:33,238 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 20:54:33,239 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:33,239 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:33,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:33,239 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:54:33,239 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:54:33,239 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:54:33,239 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:33,239 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:54:33,242 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:33,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:33,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:33,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:33,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:33,243 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:54:33,243 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:33,243 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:54:33,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:33,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:33,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:33,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:33,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:33,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:33,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:54:33,248 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:54:33,248 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:54:33,248 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:54:33,248 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:54:33,248 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:54:33,248 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:54:33,250 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:33,250 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:33,250 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:33,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:33,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:33,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:33,250 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:33,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:33,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:33,250 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:33,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:33,250 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:33,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:33,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:33,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:33,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:33,253 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:33,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:33,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:33,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:33,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:33,259 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:33,259 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:33,259 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:33,259 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:33,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:33,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:33,259 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:54:33,259 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:54:33,259 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:33,259 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:33,259 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:33,259 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:33,259 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:33,259 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:33,259 DEBUG [Thread-837] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:54:33,259 DEBUG [Thread-837] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:54:33,260 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:33,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:33,260 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:54:33,261 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304192fdf6f395a604d09b95e7ac1fba8df12 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304192fdf6f395a604d09b95e7ac1fba8df12 2023-04-19 20:54:33,273 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:33,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56352 deadline: 1681937733270, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:33,274 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681937673255, max timestamp=1681937673272Num uniques 859; 2023-04-19 20:54:33,274 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-19 20:54:33,274 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681937673255, max timestamp=1681937673272Num uniques 859; 2023-04-19 20:54:33,274 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-19 20:54:33,274 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681937673255, max timestamp=1681937673272Num uniques 859; 2023-04-19 20:54:33,274 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-19 20:54:33,279 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=395, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/A/47cbb6a78d5d47ec87b0c66df5b5771f 2023-04-19 20:54:33,305 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=395 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/B/1ba67e80777043219727014b9e100ada 2023-04-19 20:54:33,313 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:33,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56368 deadline: 1681937733308, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:33,314 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:33,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56326 deadline: 1681937733309, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:33,327 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=395 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/C/afb382a30225400fafa01dc2ed137da9 2023-04-19 20:54:33,331 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/A/47cbb6a78d5d47ec87b0c66df5b5771f as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/47cbb6a78d5d47ec87b0c66df5b5771f 2023-04-19 20:54:33,335 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/47cbb6a78d5d47ec87b0c66df5b5771f, entries=350, sequenceid=395, filesize=52.9 K 2023-04-19 20:54:33,336 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/B/1ba67e80777043219727014b9e100ada as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/1ba67e80777043219727014b9e100ada 2023-04-19 20:54:33,341 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/1ba67e80777043219727014b9e100ada, entries=150, sequenceid=395, filesize=11.9 K 2023-04-19 20:54:33,342 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/C/afb382a30225400fafa01dc2ed137da9 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/afb382a30225400fafa01dc2ed137da9 2023-04-19 20:54:33,356 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/afb382a30225400fafa01dc2ed137da9, entries=150, sequenceid=395, filesize=11.9 K 2023-04-19 20:54:33,357 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~71.56 KB/73280, heapSize ~190.34 KB/194912, currentSize=120.76 KB/123660 for c7ee8087efe9c3ae02af767e46864c85 in 121ms, sequenceid=395, compaction requested=true 2023-04-19 20:54:33,357 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c7ee8087efe9c3ae02af767e46864c85: 2023-04-19 20:54:33,357 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 20:54:33,358 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-19 20:54:33,358 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=3), splitQueue=0 2023-04-19 20:54:33,360 DEBUG [Thread-837] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:54:33,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:54:33,363 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:54:33,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:54:33,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:54:33,365 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:54:33,365 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:33,365 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:54:33,365 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:54:33,365 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:33,365 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:33,367 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:33,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:33,367 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:33,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:33,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:33,367 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:33,367 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:54:33,368 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:33,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:54:33,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:33,368 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:54:33,372 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:54:33,372 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:54:33,372 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:54:33,372 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:33,372 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:54:33,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(9158): Flush requested on c7ee8087efe9c3ae02af767e46864c85 2023-04-19 20:54:33,377 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c7ee8087efe9c3ae02af767e46864c85 3/3 column families, dataSize=127.47 KB heapSize=360.38 KB 2023-04-19 20:54:33,377 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c7ee8087efe9c3ae02af767e46864c85, store=A 2023-04-19 20:54:33,377 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 20:54:33,377 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c7ee8087efe9c3ae02af767e46864c85, store=B 2023-04-19 20:54:33,377 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 20:54:33,377 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c7ee8087efe9c3ae02af767e46864c85, store=C 2023-04-19 20:54:33,377 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 20:54:33,381 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:54:33,382 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:54:33,384 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:33,385 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. started... 2023-04-19 20:54:33,385 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. as already flushing 2023-04-19 20:54:33,385 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:33,385 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:54:33,385 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:54:33,385 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:33,385 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:54:33,385 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:54:33,387 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:33,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:33,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:33,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:33,387 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:33,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:33,388 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:33,388 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:54:33,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:33,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:33,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:33,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:33,389 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:54:33,389 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6738c012[Count = 0] remaining members to acquire global barrier 2023-04-19 20:54:33,389 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:54:33,389 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:33,392 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:33,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:33,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:33,392 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:54:33,392 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:54:33,392 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:54:33,392 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:33,392 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:54:33,394 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:54:33,394 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:33,394 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:54:33,394 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:33,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:33,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:33,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:33,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:33,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:33,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:33,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:33,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:33,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:33,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:33,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:54:33,398 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:54:33,398 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:54:33,398 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:54:33,398 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:54:33,398 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:54:33,399 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:54:33,400 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:33,400 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:33,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:33,400 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:33,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:33,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:33,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:33,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:33,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:33,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:33,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:33,401 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:33,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:33,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:33,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:33,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:33,402 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:33,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:33,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:33,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:33,405 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:33,405 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:33,405 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:33,405 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:33,405 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:33,405 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:54:33,405 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:54:33,405 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:33,405 DEBUG [Thread-837] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:54:33,405 DEBUG [Thread-837] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:54:33,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:33,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:33,405 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:33,405 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:33,405 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:33,405 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:33,406 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:54:33,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:33,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:33,424 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:33,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56368 deadline: 1681937733416, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:33,429 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:33,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56352 deadline: 1681937733423, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:33,429 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:33,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56326 deadline: 1681937733424, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:33,505 DEBUG [Thread-837] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:54:33,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:54:33,508 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:54:33,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:54:33,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:54:33,509 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:54:33,509 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:33,509 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:54:33,509 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:54:33,510 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:33,510 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:33,512 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:33,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:33,512 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:33,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:33,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:33,513 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:33,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:54:33,513 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:33,513 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:54:33,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:33,513 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:54:33,514 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:54:33,514 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:54:33,514 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:54:33,514 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:33,514 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:54:33,514 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:54:33,514 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:54:33,514 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:33,514 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. started... 2023-04-19 20:54:33,514 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. as already flushing 2023-04-19 20:54:33,514 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:33,514 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:54:33,514 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:54:33,514 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:33,514 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:54:33,514 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:54:33,516 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:33,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:33,516 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:33,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:33,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:33,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:33,516 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:33,516 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:54:33,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:33,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:33,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:33,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:33,518 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:54:33,518 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@484f4a7f[Count = 0] remaining members to acquire global barrier 2023-04-19 20:54:33,518 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:54:33,518 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:33,519 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:33,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:33,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:33,519 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:54:33,519 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:54:33,519 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:54:33,519 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:33,519 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:54:33,521 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:33,521 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:54:33,521 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:33,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:33,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:33,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:33,521 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:54:33,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:33,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:33,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:33,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:33,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:33,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:33,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:33,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:54:33,524 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:54:33,524 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:54:33,524 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:54:33,524 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:54:33,524 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:54:33,524 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:54:33,527 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:33,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:33,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:33,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:33,529 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:33,529 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:33,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:33,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:33,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:33,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:33,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:33,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:33,531 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:33,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:33,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:33,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:33,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:33,532 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:33,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:33,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:33,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:33,533 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:33,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56326 deadline: 1681937733530, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:33,533 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:33,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56352 deadline: 1681937733530, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:33,535 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:33,535 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:54:33,535 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:54:33,535 DEBUG [Thread-837] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:54:33,535 DEBUG [Thread-837] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:54:33,536 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:33,536 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:33,536 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:33,536 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:33,536 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:33,536 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:33,536 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:33,536 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:33,536 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:33,536 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:33,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:33,536 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:33,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:33,621 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/A/60939d5542bb409f82114fd3bcdf1811 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/60939d5542bb409f82114fd3bcdf1811 2023-04-19 20:54:33,626 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in c7ee8087efe9c3ae02af767e46864c85/A of c7ee8087efe9c3ae02af767e46864c85 into 60939d5542bb409f82114fd3bcdf1811(size=26.1 K), total size for store is 129.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 20:54:33,636 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c7ee8087efe9c3ae02af767e46864c85: 2023-04-19 20:54:33,636 DEBUG [Thread-837] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:54:33,637 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85., storeName=c7ee8087efe9c3ae02af767e46864c85/A, priority=12, startTime=1681937672714; duration=0sec 2023-04-19 20:54:33,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:54:33,637 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=3), splitQueue=0 2023-04-19 20:54:33,637 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 5 compacting, 2 eligible, 16 blocking 2023-04-19 20:54:33,641 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-19 20:54:33,641 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-19 20:54:33,641 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. because compaction request was cancelled 2023-04-19 20:54:33,641 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-19 20:54:33,641 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:54:33,642 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:33,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:54:33,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56368 deadline: 1681937733626, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:33,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:54:33,643 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:54:33,643 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:33,643 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:54:33,643 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:54:33,643 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:33,643 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:33,644 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 132996 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-19 20:54:33,644 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1912): c7ee8087efe9c3ae02af767e46864c85/A is initiating minor compaction (all files) 2023-04-19 20:54:33,644 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/C/f0c33174d9684abda8f058a9efbe43fc as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/f0c33174d9684abda8f058a9efbe43fc 2023-04-19 20:54:33,644 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c7ee8087efe9c3ae02af767e46864c85/A in TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:33,645 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:33,645 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:33,645 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/60939d5542bb409f82114fd3bcdf1811, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/2777891c51d64a84a3e54bfb01fc4a0f, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/f9fa9e3da88c43a6995013c0c70482ed, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/47cbb6a78d5d47ec87b0c66df5b5771f] into tmpdir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp, totalSize=129.9 K 2023-04-19 20:54:33,647 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:33,647 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:54:33,645 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:33,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:33,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:33,648 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 60939d5542bb409f82114fd3bcdf1811, keycount=150, bloomtype=ROW, size=26.1 K, encoding=NONE, compression=NONE, seqNum=335, earliestPutTs=1681937671523 2023-04-19 20:54:33,648 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:33,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:54:33,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:33,649 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:54:33,649 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:54:33,649 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:54:33,649 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:54:33,649 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:33,649 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:54:33,651 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 2777891c51d64a84a3e54bfb01fc4a0f, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=345, earliestPutTs=1681937671826 2023-04-19 20:54:33,652 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting f9fa9e3da88c43a6995013c0c70482ed, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=351, earliestPutTs=1681937672999 2023-04-19 20:54:33,652 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:54:33,652 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:54:33,652 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:33,652 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. started... 2023-04-19 20:54:33,652 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. as already flushing 2023-04-19 20:54:33,652 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 47cbb6a78d5d47ec87b0c66df5b5771f, keycount=350, bloomtype=ROW, size=52.9 K, encoding=NONE, compression=NONE, seqNum=395, earliestPutTs=1681937673213 2023-04-19 20:54:33,652 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:33,652 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:54:33,653 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:54:33,653 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:33,653 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:54:33,653 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:54:33,654 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in c7ee8087efe9c3ae02af767e46864c85/C of c7ee8087efe9c3ae02af767e46864c85 into f0c33174d9684abda8f058a9efbe43fc(size=12.6 K), total size for store is 34.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 20:54:33,654 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c7ee8087efe9c3ae02af767e46864c85: 2023-04-19 20:54:33,654 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85., storeName=c7ee8087efe9c3ae02af767e46864c85/C, priority=11, startTime=1681937672714; duration=0sec 2023-04-19 20:54:33,654 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 20:54:33,654 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-19 20:54:33,660 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:33,662 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:33,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56408 deadline: 1681937733657, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:33,662 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:33,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:33,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:33,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:33,662 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:33,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56384 deadline: 1681937733658, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:33,662 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:33,662 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:54:33,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:33,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:33,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:33,664 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 46948 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-19 20:54:33,664 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1912): c7ee8087efe9c3ae02af767e46864c85/B is initiating minor compaction (all files) 2023-04-19 20:54:33,664 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c7ee8087efe9c3ae02af767e46864c85/B in TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:33,664 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/8f53f8ac6c6c4962a8475ec89300ef67, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/c8caf13fe1594064bf33ee22afa5d125, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/8aa9018bc5b04eab9765f5b3b73b5244, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/1ba67e80777043219727014b9e100ada] into tmpdir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp, totalSize=45.8 K 2023-04-19 20:54:33,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:33,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:33,665 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 8f53f8ac6c6c4962a8475ec89300ef67, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=335, earliestPutTs=1681937671523 2023-04-19 20:54:33,665 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:54:33,666 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@12ee63f5[Count = 0] remaining members to acquire global barrier 2023-04-19 20:54:33,666 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:54:33,666 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:33,666 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting c8caf13fe1594064bf33ee22afa5d125, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=345, earliestPutTs=1681937671826 2023-04-19 20:54:33,666 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 8aa9018bc5b04eab9765f5b3b73b5244, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=351, earliestPutTs=1681937673025 2023-04-19 20:54:33,667 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 1ba67e80777043219727014b9e100ada, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=395, earliestPutTs=1681937673224 2023-04-19 20:54:33,668 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:33,668 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:33,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:33,668 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:54:33,669 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:54:33,669 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:54:33,669 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:33,669 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:54:33,670 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:54:33,670 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:33,670 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:54:33,670 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:33,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:33,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:33,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:33,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:33,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:33,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:33,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:33,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:33,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:33,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:33,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:54:33,675 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:54:33,675 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:54:33,675 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:54:33,675 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:54:33,675 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:54:33,675 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:54:33,680 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:33,680 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:33,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:33,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:33,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:33,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:33,681 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:33,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:33,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:33,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:33,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:33,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:33,682 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:33,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:33,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:33,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:33,684 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:33,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:33,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:33,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:33,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:33,689 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:33,689 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:33,689 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:33,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:33,689 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:33,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:33,689 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:33,690 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:33,690 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:33,690 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:33,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:33,690 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:33,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:33,690 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:33,690 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:54:33,690 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:54:33,690 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:54:33,690 DEBUG [Thread-837] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:54:33,690 DEBUG [Thread-837] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:54:33,698 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] throttle.PressureAwareThroughputController(145): c7ee8087efe9c3ae02af767e46864c85#B#compaction#176 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-19 20:54:33,701 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] throttle.PressureAwareThroughputController(145): c7ee8087efe9c3ae02af767e46864c85#A#compaction#175 average throughput is 0.90 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 20:54:33,730 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/B/6888c6d185f940c2a9d741405f776996 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/6888c6d185f940c2a9d741405f776996 2023-04-19 20:54:33,737 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in c7ee8087efe9c3ae02af767e46864c85/B of c7ee8087efe9c3ae02af767e46864c85 into 6888c6d185f940c2a9d741405f776996(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 20:54:33,737 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c7ee8087efe9c3ae02af767e46864c85: 2023-04-19 20:54:33,737 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85., storeName=c7ee8087efe9c3ae02af767e46864c85/B, priority=12, startTime=1681937673357; duration=0sec 2023-04-19 20:54:33,737 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 20:54:33,739 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:33,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56326 deadline: 1681937733735, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:33,739 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:33,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56352 deadline: 1681937733736, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:33,764 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/A/c765a0f5b6ba48998721ef5e2dfc3b7c as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/c765a0f5b6ba48998721ef5e2dfc3b7c 2023-04-19 20:54:33,770 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in c7ee8087efe9c3ae02af767e46864c85/A of c7ee8087efe9c3ae02af767e46864c85 into c765a0f5b6ba48998721ef5e2dfc3b7c(size=26.2 K), total size for store is 26.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 20:54:33,770 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c7ee8087efe9c3ae02af767e46864c85: 2023-04-19 20:54:33,770 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85., storeName=c7ee8087efe9c3ae02af767e46864c85/A, priority=12, startTime=1681937673357; duration=0sec 2023-04-19 20:54:33,770 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 20:54:33,791 DEBUG [Thread-837] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:54:33,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:54:33,793 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:54:33,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:54:33,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:54:33,795 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:54:33,795 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:33,795 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:54:33,795 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:54:33,795 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:33,795 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:33,797 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:33,797 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:33,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:33,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:33,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:33,797 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:33,797 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:54:33,797 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:33,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:54:33,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:33,798 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:54:33,798 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:54:33,798 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:54:33,798 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:54:33,798 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:33,798 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:54:33,804 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:54:33,804 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:54:33,804 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:33,804 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. started... 2023-04-19 20:54:33,804 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. as already flushing 2023-04-19 20:54:33,804 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:33,804 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:54:33,804 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:54:33,804 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:33,804 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:54:33,804 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:54:33,808 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:33,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:33,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:33,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:33,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:33,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:33,809 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:33,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:33,809 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:33,809 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:54:33,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:33,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:33,810 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:54:33,810 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@66984bc7[Count = 0] remaining members to acquire global barrier 2023-04-19 20:54:33,810 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:54:33,810 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:33,812 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:33,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:33,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:33,812 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:54:33,812 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:54:33,812 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:54:33,812 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:33,813 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:54:33,814 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:33,814 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:54:33,814 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:33,814 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:54:33,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:33,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:33,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:33,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:33,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:33,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:33,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:33,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:33,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:33,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:33,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:54:33,817 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:54:33,817 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:54:33,817 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:54:33,817 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:54:33,817 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:54:33,817 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:54:33,819 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:33,819 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:33,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:33,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:33,819 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:33,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:33,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:33,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:33,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:33,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:33,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:33,820 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:33,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:33,822 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304190e95a73ac6f74ef6aa9bfe47c87e113a to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304190e95a73ac6f74ef6aa9bfe47c87e113a 2023-04-19 20:54:33,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:33,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:33,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:33,829 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:33,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:33,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:33,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:33,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:33,832 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:33,832 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:33,832 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:33,832 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:33,832 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:33,832 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:33,832 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:33,833 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:33,833 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:33,833 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:33,833 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:54:33,833 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:54:33,833 DEBUG [Thread-837] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:54:33,833 DEBUG [Thread-837] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:54:33,833 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:33,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:33,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:33,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:33,834 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:54:33,850 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=430, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/A/82bc10ae78dd4c178e79e3cb8d48d793 2023-04-19 20:54:33,878 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=430 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/B/f649d570960f47519c41ae06824b0d95 2023-04-19 20:54:33,899 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=430 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/C/27f688ef68364a819ae0f0ffce6000f3 2023-04-19 20:54:33,905 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/A/82bc10ae78dd4c178e79e3cb8d48d793 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/82bc10ae78dd4c178e79e3cb8d48d793 2023-04-19 20:54:33,909 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/82bc10ae78dd4c178e79e3cb8d48d793, entries=200, sequenceid=430, filesize=32.3 K 2023-04-19 20:54:33,910 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/B/f649d570960f47519c41ae06824b0d95 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/f649d570960f47519c41ae06824b0d95 2023-04-19 20:54:33,925 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/f649d570960f47519c41ae06824b0d95, entries=150, sequenceid=430, filesize=11.9 K 2023-04-19 20:54:33,926 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/C/27f688ef68364a819ae0f0ffce6000f3 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/27f688ef68364a819ae0f0ffce6000f3 2023-04-19 20:54:33,933 DEBUG [Thread-837] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:54:33,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:54:33,936 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:54:33,939 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/27f688ef68364a819ae0f0ffce6000f3, entries=150, sequenceid=430, filesize=11.9 K 2023-04-19 20:54:33,940 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~360.33 KB/368976, currentSize=53.67 KB/54960 for c7ee8087efe9c3ae02af767e46864c85 in 563ms, sequenceid=430, compaction requested=true 2023-04-19 20:54:33,940 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c7ee8087efe9c3ae02af767e46864c85: 2023-04-19 20:54:33,940 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-04-19 20:54:33,940 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 20:54:33,940 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-04-19 20:54:33,941 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 20:54:33,941 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 20:54:33,941 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-19 20:54:33,941 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-19 20:54:33,941 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. because compaction request was cancelled 2023-04-19 20:54:33,941 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-19 20:54:33,941 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-19 20:54:33,941 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-19 20:54:33,941 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. because compaction request was cancelled 2023-04-19 20:54:33,942 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 46982 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-19 20:54:33,943 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1912): c7ee8087efe9c3ae02af767e46864c85/C is initiating minor compaction (all files) 2023-04-19 20:54:33,943 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c7ee8087efe9c3ae02af767e46864c85/C in TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:33,943 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/f0c33174d9684abda8f058a9efbe43fc, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/873cb6ed3c2245e7a75fc47062026599, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/afb382a30225400fafa01dc2ed137da9, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/27f688ef68364a819ae0f0ffce6000f3] into tmpdir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp, totalSize=45.9 K 2023-04-19 20:54:33,943 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting f0c33174d9684abda8f058a9efbe43fc, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=345, earliestPutTs=1681937671826 2023-04-19 20:54:33,943 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 873cb6ed3c2245e7a75fc47062026599, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=351, earliestPutTs=1681937673025 2023-04-19 20:54:33,944 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting afb382a30225400fafa01dc2ed137da9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=395, earliestPutTs=1681937673224 2023-04-19 20:54:33,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:54:33,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:54:33,944 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:54:33,944 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 27f688ef68364a819ae0f0ffce6000f3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=430, earliestPutTs=1681937673306 2023-04-19 20:54:33,944 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:33,944 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:54:33,944 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:54:33,945 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:33,945 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:33,955 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:33,955 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:33,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:33,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:33,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:33,956 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:33,956 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:54:33,956 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:33,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:54:33,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:33,957 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:54:33,957 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:54:33,958 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:54:33,958 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:54:33,958 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:33,958 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:54:33,958 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:54:33,958 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:54:33,958 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:33,958 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. started... 2023-04-19 20:54:33,958 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c7ee8087efe9c3ae02af767e46864c85 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-19 20:54:33,963 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] throttle.PressureAwareThroughputController(145): c7ee8087efe9c3ae02af767e46864c85#C#compaction#179 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 20:54:33,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(9158): Flush requested on c7ee8087efe9c3ae02af767e46864c85 2023-04-19 20:54:33,972 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. as already flushing 2023-04-19 20:54:33,973 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c7ee8087efe9c3ae02af767e46864c85, store=A 2023-04-19 20:54:33,973 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:54:33,973 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c7ee8087efe9c3ae02af767e46864c85, store=B 2023-04-19 20:54:33,973 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:54:33,973 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c7ee8087efe9c3ae02af767e46864c85, store=C 2023-04-19 20:54:33,973 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:54:34,023 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/C/183283551a36445aa0a69020785d8166 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/183283551a36445aa0a69020785d8166 2023-04-19 20:54:34,038 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in c7ee8087efe9c3ae02af767e46864c85/C of c7ee8087efe9c3ae02af767e46864c85 into 183283551a36445aa0a69020785d8166(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 20:54:34,038 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c7ee8087efe9c3ae02af767e46864c85: 2023-04-19 20:54:34,038 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85., storeName=c7ee8087efe9c3ae02af767e46864c85/C, priority=12, startTime=1681937673941; duration=0sec 2023-04-19 20:54:34,039 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 20:54:34,052 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:34,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56368 deadline: 1681937734049, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:34,053 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:34,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56326 deadline: 1681937734050, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:34,057 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:34,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56352 deadline: 1681937734052, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:34,156 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:34,156 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:34,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56368 deadline: 1681937734153, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:34,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56326 deadline: 1681937734154, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:34,161 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:34,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56352 deadline: 1681937734158, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:34,360 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:34,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56326 deadline: 1681937734358, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:34,361 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:34,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56368 deadline: 1681937734359, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:34,368 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:34,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56352 deadline: 1681937734364, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:34,418 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041954a4732e23d84f3484730dbfe675e139 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041954a4732e23d84f3484730dbfe675e139 2023-04-19 20:54:34,667 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:34,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56368 deadline: 1681937734663, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:34,667 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:34,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56326 deadline: 1681937734663, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:34,674 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:34,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56352 deadline: 1681937734671, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:34,829 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=446, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/A/7ba7a588d2df48218eac186d19f16bf8 2023-04-19 20:54:35,169 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:35,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56326 deadline: 1681937735169, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:35,174 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:35,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56368 deadline: 1681937735172, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:35,179 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:35,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56352 deadline: 1681937735179, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:35,244 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=446 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/B/1e5b150f784e430b830cf6fb3fa2beb4 2023-04-19 20:54:35,255 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=446 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/C/0a034a27700d487ca3473d6de31eb19f 2023-04-19 20:54:35,260 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/A/7ba7a588d2df48218eac186d19f16bf8 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/7ba7a588d2df48218eac186d19f16bf8 2023-04-19 20:54:35,263 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/7ba7a588d2df48218eac186d19f16bf8, entries=200, sequenceid=446, filesize=32.3 K 2023-04-19 20:54:35,264 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/B/1e5b150f784e430b830cf6fb3fa2beb4 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/1e5b150f784e430b830cf6fb3fa2beb4 2023-04-19 20:54:35,268 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/1e5b150f784e430b830cf6fb3fa2beb4, entries=150, sequenceid=446, filesize=11.9 K 2023-04-19 20:54:35,269 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/C/0a034a27700d487ca3473d6de31eb19f as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/0a034a27700d487ca3473d6de31eb19f 2023-04-19 20:54:35,274 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/0a034a27700d487ca3473d6de31eb19f, entries=150, sequenceid=446, filesize=11.9 K 2023-04-19 20:54:35,275 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for c7ee8087efe9c3ae02af767e46864c85 in 1317ms, sequenceid=446, compaction requested=true 2023-04-19 20:54:35,275 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c7ee8087efe9c3ae02af767e46864c85: 2023-04-19 20:54:35,275 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:35,276 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:54:35,276 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:54:35,276 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:35,276 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:54:35,276 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:54:35,277 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:35,277 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:35,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:35,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:35,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:35,278 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:35,278 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:54:35,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:35,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:35,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:35,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:35,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:35,280 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:54:35,280 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@47013f56[Count = 0] remaining members to acquire global barrier 2023-04-19 20:54:35,280 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:54:35,280 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:35,281 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:35,281 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:35,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:35,281 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:54:35,281 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:54:35,281 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:54:35,283 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:54:35,283 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:35,283 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:54:35,283 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:35,283 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:54:35,283 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:54:35,283 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:54:35,283 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:54:35,283 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:54:35,283 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:54:35,283 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:54:35,286 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:35,286 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:35,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:35,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:35,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:35,286 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:35,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:35,286 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:35,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:35,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:35,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:35,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:35,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:35,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:35,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:35,288 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:35,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:35,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:35,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:35,289 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:35,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:35,292 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:35,292 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:35,292 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:35,292 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:35,292 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:35,292 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:35,292 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:54:35,292 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:54:35,292 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:35,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:35,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:35,292 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:35,293 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:35,293 DEBUG [Thread-837] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:54:35,292 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:35,293 DEBUG [Thread-837] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:54:35,293 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:35,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:35,393 DEBUG [Thread-837] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:54:35,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:54:35,395 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:54:35,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:54:35,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:54:35,396 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:54:35,396 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:35,396 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:54:35,397 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:54:35,397 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:35,397 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:35,398 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:35,398 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:35,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:35,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:35,399 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:35,399 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:54:35,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:35,399 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:35,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:54:35,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:35,399 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:54:35,400 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:54:35,400 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:54:35,400 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:54:35,400 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:35,400 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:54:35,401 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:54:35,401 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:54:35,401 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:35,401 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. started... 2023-04-19 20:54:35,401 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c7ee8087efe9c3ae02af767e46864c85 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-19 20:54:35,402 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c7ee8087efe9c3ae02af767e46864c85, store=A 2023-04-19 20:54:35,402 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:54:35,402 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c7ee8087efe9c3ae02af767e46864c85, store=B 2023-04-19 20:54:35,402 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:54:35,402 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c7ee8087efe9c3ae02af767e46864c85, store=C 2023-04-19 20:54:35,402 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:54:35,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(9158): Flush requested on c7ee8087efe9c3ae02af767e46864c85 2023-04-19 20:54:35,670 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. as already flushing 2023-04-19 20:54:35,684 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:35,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56384 deadline: 1681937735682, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:35,684 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:35,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56408 deadline: 1681937735682, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:35,787 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:35,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56384 deadline: 1681937735785, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:35,791 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:35,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56408 deadline: 1681937735788, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:35,845 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230419ee7607845217489fa0be7a16c4f9deb9 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419ee7607845217489fa0be7a16c4f9deb9 2023-04-19 20:54:35,992 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:35,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56384 deadline: 1681937735989, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:35,992 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:35,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56408 deadline: 1681937735992, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:36,182 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:36,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56326 deadline: 1681937736176, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:36,183 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:36,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56368 deadline: 1681937736181, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:36,183 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:36,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56352 deadline: 1681937736181, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:36,282 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=467, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/A/504ee19cfb0c41d79925848025a02888 2023-04-19 20:54:36,296 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:36,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56384 deadline: 1681937736293, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:36,296 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:36,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56408 deadline: 1681937736294, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:36,696 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=467 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/B/7b31eeb19d614cd88b6e5700bc4efcba 2023-04-19 20:54:36,710 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=467 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/C/c60fbab40ecd4f949b59e1b97c06505a 2023-04-19 20:54:36,715 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/A/504ee19cfb0c41d79925848025a02888 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/504ee19cfb0c41d79925848025a02888 2023-04-19 20:54:36,720 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/504ee19cfb0c41d79925848025a02888, entries=150, sequenceid=467, filesize=25.4 K 2023-04-19 20:54:36,721 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/B/7b31eeb19d614cd88b6e5700bc4efcba as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/7b31eeb19d614cd88b6e5700bc4efcba 2023-04-19 20:54:36,725 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/7b31eeb19d614cd88b6e5700bc4efcba, entries=150, sequenceid=467, filesize=11.9 K 2023-04-19 20:54:36,726 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/C/c60fbab40ecd4f949b59e1b97c06505a as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/c60fbab40ecd4f949b59e1b97c06505a 2023-04-19 20:54:36,730 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/c60fbab40ecd4f949b59e1b97c06505a, entries=150, sequenceid=467, filesize=11.9 K 2023-04-19 20:54:36,730 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for c7ee8087efe9c3ae02af767e46864c85 in 1329ms, sequenceid=467, compaction requested=true 2023-04-19 20:54:36,730 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c7ee8087efe9c3ae02af767e46864c85: 2023-04-19 20:54:36,731 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:36,731 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:54:36,731 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:54:36,731 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:36,731 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:54:36,731 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:54:36,732 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:36,732 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:36,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:36,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:36,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:36,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:36,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:36,734 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:36,734 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:54:36,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:36,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:36,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:36,735 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:54:36,735 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@75af997b[Count = 0] remaining members to acquire global barrier 2023-04-19 20:54:36,735 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:54:36,735 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:36,737 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:36,737 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:36,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:36,737 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:54:36,737 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:54:36,737 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:54:36,738 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:36,738 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:54:36,739 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:36,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:36,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:36,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:36,739 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:54:36,739 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:36,739 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:54:36,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:36,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:36,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:36,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:36,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:36,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:36,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:36,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:54:36,741 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:54:36,741 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:54:36,741 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:54:36,742 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:54:36,742 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:54:36,742 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:54:36,744 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:36,744 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:36,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:36,744 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:36,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:36,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:36,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:36,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:36,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:36,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:36,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:36,745 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:36,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:36,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:36,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:36,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:36,749 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:36,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:36,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:36,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:36,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:36,753 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:36,753 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:36,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:36,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:36,753 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:36,753 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:54:36,753 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:54:36,753 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:54:36,753 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:36,753 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:36,753 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:36,753 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:36,754 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:36,754 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:36,754 DEBUG [Thread-837] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:54:36,754 DEBUG [Thread-837] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:54:36,754 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:36,754 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:36,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:36,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(9158): Flush requested on c7ee8087efe9c3ae02af767e46864c85 2023-04-19 20:54:36,799 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c7ee8087efe9c3ae02af767e46864c85 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-19 20:54:36,801 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c7ee8087efe9c3ae02af767e46864c85, store=A 2023-04-19 20:54:36,801 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:54:36,801 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c7ee8087efe9c3ae02af767e46864c85, store=B 2023-04-19 20:54:36,801 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:54:36,801 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c7ee8087efe9c3ae02af767e46864c85, store=C 2023-04-19 20:54:36,801 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:54:36,821 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230419b4b9707441344bbd99a53beae40ab597 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419b4b9707441344bbd99a53beae40ab597 2023-04-19 20:54:36,837 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:36,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56384 deadline: 1681937736832, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:36,837 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:36,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56408 deadline: 1681937736833, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:36,854 DEBUG [Thread-837] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:54:36,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:54:36,856 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:54:36,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:54:36,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:54:36,857 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:54:36,857 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:36,857 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:54:36,857 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:54:36,858 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:36,858 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:36,862 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:36,862 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:36,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:36,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:36,863 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:36,863 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:54:36,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:36,863 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:36,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:54:36,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:36,864 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:54:36,864 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:54:36,864 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:54:36,864 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:54:36,864 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:36,864 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:54:36,864 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:54:36,864 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:54:36,864 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:36,864 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. started... 2023-04-19 20:54:36,864 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. as already flushing 2023-04-19 20:54:36,865 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:36,865 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:54:36,865 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:54:36,865 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:36,865 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:54:36,865 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:54:36,866 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:36,866 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:36,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:36,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:36,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:36,866 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:36,866 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:54:36,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:36,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:36,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:36,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:36,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:36,868 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:54:36,868 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@14bfbdf0[Count = 0] remaining members to acquire global barrier 2023-04-19 20:54:36,868 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:54:36,868 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:36,869 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:36,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:36,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:36,869 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:36,869 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:54:36,869 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:54:36,869 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:54:36,869 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:54:36,871 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:36,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:36,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:36,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:36,871 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:54:36,871 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:36,871 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:54:36,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:36,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:36,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:36,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:36,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:36,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:36,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:36,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:54:36,874 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:54:36,874 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:54:36,874 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:54:36,874 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:54:36,874 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:54:36,874 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:54:36,875 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:36,875 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:36,875 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:36,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:36,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:36,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:36,875 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:36,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:36,876 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:36,876 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:36,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:36,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:36,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:36,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:36,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:36,877 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:36,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:36,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:36,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:36,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:36,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:36,880 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:36,880 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:36,880 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:36,880 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:36,880 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:36,880 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:36,880 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:36,880 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:36,880 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:36,880 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:36,880 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:54:36,881 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:54:36,881 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:54:36,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:36,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:36,881 DEBUG [Thread-837] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:54:36,881 DEBUG [Thread-837] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:54:36,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:36,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:36,938 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:36,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56384 deadline: 1681937736938, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:36,939 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:36,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56408 deadline: 1681937736938, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:36,981 DEBUG [Thread-837] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:54:36,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:54:36,983 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:54:36,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:54:36,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:54:36,984 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:54:36,985 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:36,985 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:54:36,985 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:54:36,985 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:36,985 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:36,986 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:36,986 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:36,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:36,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:36,987 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:36,987 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:54:36,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:36,987 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:36,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:54:36,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:36,988 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:54:36,988 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:54:36,988 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:54:36,988 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:54:36,988 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:36,988 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:54:36,988 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:54:36,988 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:54:36,988 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:36,988 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. started... 2023-04-19 20:54:36,989 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. as already flushing 2023-04-19 20:54:36,989 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:36,989 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:54:36,989 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:54:36,989 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:36,989 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:54:36,989 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:54:36,991 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:36,991 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:36,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:36,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:36,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:36,992 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:36,992 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:54:36,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:36,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:36,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:36,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:36,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:36,993 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:54:36,993 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@305d0ad3[Count = 0] remaining members to acquire global barrier 2023-04-19 20:54:36,993 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:54:36,993 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:36,995 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:36,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:36,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:36,995 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:54:36,995 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:54:36,995 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:54:36,997 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:36,997 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:54:36,998 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:36,998 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:54:36,998 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:36,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:36,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:36,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:36,998 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:54:36,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:36,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:36,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:37,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:37,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:37,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:37,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:37,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:54:37,001 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:54:37,001 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:54:37,001 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:54:37,001 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:54:37,001 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:54:37,001 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:54:37,003 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:37,003 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:37,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:37,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:37,003 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:37,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:37,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:37,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:37,003 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:37,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:37,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:37,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:37,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:37,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:37,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:37,005 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:37,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:37,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:37,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:37,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:37,008 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:37,008 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:37,008 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:37,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:37,008 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:37,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:37,008 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:54:37,008 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:54:37,008 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:37,008 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:37,008 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:37,008 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:37,009 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:37,009 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:54:37,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:37,009 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:37,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:37,009 DEBUG [Thread-837] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:54:37,009 DEBUG [Thread-837] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:54:37,103 DEBUG [Thread-840] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4fbb4986 to 127.0.0.1:58736 2023-04-19 20:54:37,103 DEBUG [Thread-840] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-19 20:54:37,105 DEBUG [Thread-844] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x40883381 to 127.0.0.1:58736 2023-04-19 20:54:37,105 DEBUG [Thread-844] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-19 20:54:37,105 DEBUG [Thread-838] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3bccfdc5 to 127.0.0.1:58736 2023-04-19 20:54:37,105 DEBUG [Thread-838] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-19 20:54:37,107 DEBUG [Thread-842] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0b69850d to 127.0.0.1:58736 2023-04-19 20:54:37,107 DEBUG [Thread-842] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-19 20:54:37,109 DEBUG [Thread-837] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:54:37,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:54:37,141 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:37,141 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:37,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56408 deadline: 1681937737141, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:37,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56384 deadline: 1681937737141, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:37,229 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=481, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/A/4965da34965e4f8e968be6721650221d 2023-04-19 20:54:37,443 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:37,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56408 deadline: 1681937737443, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:37,443 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:37,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56384 deadline: 1681937737443, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:37,638 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=481 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/B/a56cd1e8eb3b413483cd2ad3a5851687 2023-04-19 20:54:37,946 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:37,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56384 deadline: 1681937737946, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:37,948 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:37,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56408 deadline: 1681937737948, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:38,048 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=481 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/C/c44065b7e2204744b671616cdcfdf04d 2023-04-19 20:54:38,052 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/A/4965da34965e4f8e968be6721650221d as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/4965da34965e4f8e968be6721650221d 2023-04-19 20:54:38,055 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/4965da34965e4f8e968be6721650221d, entries=150, sequenceid=481, filesize=25.4 K 2023-04-19 20:54:38,056 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/B/a56cd1e8eb3b413483cd2ad3a5851687 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/a56cd1e8eb3b413483cd2ad3a5851687 2023-04-19 20:54:38,060 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/a56cd1e8eb3b413483cd2ad3a5851687, entries=150, sequenceid=481, filesize=11.9 K 2023-04-19 20:54:38,060 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/C/c44065b7e2204744b671616cdcfdf04d as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/c44065b7e2204744b671616cdcfdf04d 2023-04-19 20:54:38,064 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/c44065b7e2204744b671616cdcfdf04d, entries=150, sequenceid=481, filesize=11.9 K 2023-04-19 20:54:38,065 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for c7ee8087efe9c3ae02af767e46864c85 in 1266ms, sequenceid=481, compaction requested=true 2023-04-19 20:54:38,065 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c7ee8087efe9c3ae02af767e46864c85: 2023-04-19 20:54:38,065 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 20:54:38,065 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 20:54:38,065 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-19 20:54:38,065 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-19 20:54:38,065 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 20:54:38,066 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 145094 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-19 20:54:38,066 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61714 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-19 20:54:38,066 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1912): c7ee8087efe9c3ae02af767e46864c85/A is initiating minor compaction (all files) 2023-04-19 20:54:38,066 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1912): c7ee8087efe9c3ae02af767e46864c85/B is initiating minor compaction (all files) 2023-04-19 20:54:38,067 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c7ee8087efe9c3ae02af767e46864c85/A in TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:38,067 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/c765a0f5b6ba48998721ef5e2dfc3b7c, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/82bc10ae78dd4c178e79e3cb8d48d793, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/7ba7a588d2df48218eac186d19f16bf8, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/504ee19cfb0c41d79925848025a02888, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/4965da34965e4f8e968be6721650221d] into tmpdir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp, totalSize=141.7 K 2023-04-19 20:54:38,067 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c7ee8087efe9c3ae02af767e46864c85/B in TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:38,067 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/6888c6d185f940c2a9d741405f776996, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/f649d570960f47519c41ae06824b0d95, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/1e5b150f784e430b830cf6fb3fa2beb4, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/7b31eeb19d614cd88b6e5700bc4efcba, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/a56cd1e8eb3b413483cd2ad3a5851687] into tmpdir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp, totalSize=60.3 K 2023-04-19 20:54:38,067 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting c765a0f5b6ba48998721ef5e2dfc3b7c, keycount=150, bloomtype=ROW, size=26.2 K, encoding=NONE, compression=NONE, seqNum=395, earliestPutTs=1681937673224 2023-04-19 20:54:38,067 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 6888c6d185f940c2a9d741405f776996, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=395, earliestPutTs=1681937673224 2023-04-19 20:54:38,067 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 82bc10ae78dd4c178e79e3cb8d48d793, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=430, earliestPutTs=1681937673306 2023-04-19 20:54:38,067 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting f649d570960f47519c41ae06824b0d95, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=430, earliestPutTs=1681937673306 2023-04-19 20:54:38,068 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 7ba7a588d2df48218eac186d19f16bf8, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=446, earliestPutTs=1681937673407 2023-04-19 20:54:38,068 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 1e5b150f784e430b830cf6fb3fa2beb4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=446, earliestPutTs=1681937673407 2023-04-19 20:54:38,068 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 504ee19cfb0c41d79925848025a02888, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=467, earliestPutTs=1681937674048 2023-04-19 20:54:38,068 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 7b31eeb19d614cd88b6e5700bc4efcba, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=467, earliestPutTs=1681937674048 2023-04-19 20:54:38,068 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 4965da34965e4f8e968be6721650221d, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=481, earliestPutTs=1681937675681 2023-04-19 20:54:38,069 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting a56cd1e8eb3b413483cd2ad3a5851687, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=481, earliestPutTs=1681937675681 2023-04-19 20:54:38,080 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] throttle.PressureAwareThroughputController(145): c7ee8087efe9c3ae02af767e46864c85#B#compaction#190 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-19 20:54:38,086 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] throttle.PressureAwareThroughputController(145): c7ee8087efe9c3ae02af767e46864c85#A#compaction#189 average throughput is 2.82 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 20:54:38,095 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/B/3cbfbf56d662491f8345cc371ed23e56 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/3cbfbf56d662491f8345cc371ed23e56 2023-04-19 20:54:38,099 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in c7ee8087efe9c3ae02af767e46864c85/B of c7ee8087efe9c3ae02af767e46864c85 into 3cbfbf56d662491f8345cc371ed23e56(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 20:54:38,099 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c7ee8087efe9c3ae02af767e46864c85: 2023-04-19 20:54:38,099 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85., storeName=c7ee8087efe9c3ae02af767e46864c85/B, priority=11, startTime=1681937678065; duration=0sec 2023-04-19 20:54:38,099 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 20:54:38,100 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-19 20:54:38,101 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49562 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-19 20:54:38,101 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1912): c7ee8087efe9c3ae02af767e46864c85/C is initiating minor compaction (all files) 2023-04-19 20:54:38,101 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c7ee8087efe9c3ae02af767e46864c85/C in TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:38,101 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/183283551a36445aa0a69020785d8166, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/0a034a27700d487ca3473d6de31eb19f, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/c60fbab40ecd4f949b59e1b97c06505a, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/c44065b7e2204744b671616cdcfdf04d] into tmpdir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp, totalSize=48.4 K 2023-04-19 20:54:38,101 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 183283551a36445aa0a69020785d8166, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=430, earliestPutTs=1681937673306 2023-04-19 20:54:38,102 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 0a034a27700d487ca3473d6de31eb19f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=446, earliestPutTs=1681937673407 2023-04-19 20:54:38,102 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting c60fbab40ecd4f949b59e1b97c06505a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=467, earliestPutTs=1681937674048 2023-04-19 20:54:38,102 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting c44065b7e2204744b671616cdcfdf04d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=481, earliestPutTs=1681937675681 2023-04-19 20:54:38,109 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] throttle.PressureAwareThroughputController(145): c7ee8087efe9c3ae02af767e46864c85#C#compaction#191 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 20:54:38,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(9158): Flush requested on c7ee8087efe9c3ae02af767e46864c85 2023-04-19 20:54:38,192 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c7ee8087efe9c3ae02af767e46864c85 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-19 20:54:38,193 DEBUG [Thread-829] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x16323966 to 127.0.0.1:58736 2023-04-19 20:54:38,193 DEBUG [Thread-829] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-19 20:54:38,193 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c7ee8087efe9c3ae02af767e46864c85, store=A 2023-04-19 20:54:38,193 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:54:38,193 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c7ee8087efe9c3ae02af767e46864c85, store=B 2023-04-19 20:54:38,193 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:54:38,193 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c7ee8087efe9c3ae02af767e46864c85, store=C 2023-04-19 20:54:38,193 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:54:38,200 DEBUG [Thread-827] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5c508332 to 127.0.0.1:58736 2023-04-19 20:54:38,200 DEBUG [Thread-827] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-19 20:54:38,202 DEBUG [Thread-835] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2a5a81b5 to 127.0.0.1:58736 2023-04-19 20:54:38,202 DEBUG [Thread-835] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-19 20:54:38,519 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/C/289896eb506f4d01b21fa5dfc8181351 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/289896eb506f4d01b21fa5dfc8181351 2023-04-19 20:54:38,523 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in c7ee8087efe9c3ae02af767e46864c85/C of c7ee8087efe9c3ae02af767e46864c85 into 289896eb506f4d01b21fa5dfc8181351(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 20:54:38,523 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c7ee8087efe9c3ae02af767e46864c85: 2023-04-19 20:54:38,523 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85., storeName=c7ee8087efe9c3ae02af767e46864c85/C, priority=12, startTime=1681937678065; duration=0sec 2023-04-19 20:54:38,523 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 20:54:38,605 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230419aea8563ede744918b8e2f63546d53800 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419aea8563ede744918b8e2f63546d53800 2023-04-19 20:54:38,948 DEBUG [Thread-831] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x52f29016 to 127.0.0.1:58736 2023-04-19 20:54:38,948 DEBUG [Thread-831] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-19 20:54:38,953 DEBUG [Thread-833] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x40c57efc to 127.0.0.1:58736 2023-04-19 20:54:38,953 DEBUG [Thread-833] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-19 20:54:38,953 INFO [Listener at localhost/34893] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-19 20:54:38,953 INFO [Listener at localhost/34893] hbase.AcidGuaranteesTestTool(394): wrote 97 2023-04-19 20:54:38,953 INFO [Listener at localhost/34893] hbase.AcidGuaranteesTestTool(394): wrote 88 2023-04-19 20:54:38,953 INFO [Listener at localhost/34893] hbase.AcidGuaranteesTestTool(394): wrote 67 2023-04-19 20:54:38,953 INFO [Listener at localhost/34893] hbase.AcidGuaranteesTestTool(394): wrote 62 2023-04-19 20:54:38,953 INFO [Listener at localhost/34893] hbase.AcidGuaranteesTestTool(394): wrote 96 2023-04-19 20:54:38,953 INFO [Listener at localhost/34893] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-19 20:54:38,953 INFO [Listener at localhost/34893] hbase.AcidGuaranteesTestTool(398): read 4867 2023-04-19 20:54:38,954 INFO [Listener at localhost/34893] hbase.AcidGuaranteesTestTool(398): read 4666 2023-04-19 20:54:38,954 INFO [Listener at localhost/34893] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-19 20:54:38,954 INFO [Listener at localhost/34893] hbase.AcidGuaranteesTestTool(402): scanned 1985 2023-04-19 20:54:38,954 INFO [Listener at localhost/34893] hbase.AcidGuaranteesTestTool(403): verified 5953 rows 2023-04-19 20:54:38,954 INFO [Listener at localhost/34893] hbase.AcidGuaranteesTestTool(402): scanned 1981 2023-04-19 20:54:38,954 INFO [Listener at localhost/34893] hbase.AcidGuaranteesTestTool(403): verified 5943 rows 2023-04-19 20:54:38,954 INFO [Listener at localhost/34893] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-19 20:54:38,954 DEBUG [Listener at localhost/34893] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x237d5550 to 127.0.0.1:58736 2023-04-19 20:54:38,954 DEBUG [Listener at localhost/34893] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-19 20:54:38,957 INFO [Listener at localhost/34893] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-19 20:54:38,957 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-19 20:54:38,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure2.ProcedureExecutor(1029): Stored pid=24, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-19 20:54:38,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-19 20:54:38,962 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681937678962"}]},"ts":"1681937678962"} 2023-04-19 20:54:38,963 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-19 20:54:38,966 INFO [PEWorker-5] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-19 20:54:38,966 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=25, ppid=24, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=c7ee8087efe9c3ae02af767e46864c85, UNASSIGN}] 2023-04-19 20:54:38,967 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=25, ppid=24, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=c7ee8087efe9c3ae02af767e46864c85, UNASSIGN 2023-04-19 20:54:38,968 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=c7ee8087efe9c3ae02af767e46864c85, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:38,968 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681937678968"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681937678968"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681937678968"}]},"ts":"1681937678968"} 2023-04-19 20:54:38,969 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=26, ppid=25, state=RUNNABLE; CloseRegionProcedure c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971}] 2023-04-19 20:54:39,010 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=501, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/A/c03bbc3cb0c54098b1d572145bb5bb37 2023-04-19 20:54:39,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-19 20:54:39,121 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close c7ee8087efe9c3ae02af767e46864c85 2023-04-19 20:54:39,121 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing c7ee8087efe9c3ae02af767e46864c85, disabling compactions & flushes 2023-04-19 20:54:39,121 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1858): waiting for 1 compactions & cache flush to complete for region TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:39,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-19 20:54:39,307 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/A/c877773aa7ec4583ba2401ea4847c9bd as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/c877773aa7ec4583ba2401ea4847c9bd 2023-04-19 20:54:39,312 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in c7ee8087efe9c3ae02af767e46864c85/A of c7ee8087efe9c3ae02af767e46864c85 into c877773aa7ec4583ba2401ea4847c9bd(size=26.4 K), total size for store is 26.4 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-19 20:54:39,312 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c7ee8087efe9c3ae02af767e46864c85: 2023-04-19 20:54:39,312 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1858): waiting for 0 compactions & cache flush to complete for region TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:39,312 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85., storeName=c7ee8087efe9c3ae02af767e46864c85/A, priority=11, startTime=1681937678065; duration=1sec 2023-04-19 20:54:39,312 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 20:54:39,421 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=501 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/B/36296c4afdc04b0297d49b30ee908166 2023-04-19 20:54:39,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-19 20:54:39,831 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=501 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/C/258956a6880843759da6e10706e85634 2023-04-19 20:54:39,835 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/A/c03bbc3cb0c54098b1d572145bb5bb37 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/c03bbc3cb0c54098b1d572145bb5bb37 2023-04-19 20:54:39,838 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/c03bbc3cb0c54098b1d572145bb5bb37, entries=150, sequenceid=501, filesize=25.4 K 2023-04-19 20:54:39,838 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/B/36296c4afdc04b0297d49b30ee908166 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/36296c4afdc04b0297d49b30ee908166 2023-04-19 20:54:39,842 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/36296c4afdc04b0297d49b30ee908166, entries=150, sequenceid=501, filesize=11.9 K 2023-04-19 20:54:39,843 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/C/258956a6880843759da6e10706e85634 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/258956a6880843759da6e10706e85634 2023-04-19 20:54:39,847 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/258956a6880843759da6e10706e85634, entries=150, sequenceid=501, filesize=11.9 K 2023-04-19 20:54:39,847 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=26.84 KB/27480 for c7ee8087efe9c3ae02af767e46864c85 in 1655ms, sequenceid=501, compaction requested=false 2023-04-19 20:54:39,847 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c7ee8087efe9c3ae02af767e46864c85: 2023-04-19 20:54:39,847 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:39,847 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:39,847 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. after waiting 0 ms 2023-04-19 20:54:39,847 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:39,848 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing c7ee8087efe9c3ae02af767e46864c85 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-19 20:54:39,848 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c7ee8087efe9c3ae02af767e46864c85, store=A 2023-04-19 20:54:39,848 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:54:39,848 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c7ee8087efe9c3ae02af767e46864c85, store=B 2023-04-19 20:54:39,848 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:54:39,848 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c7ee8087efe9c3ae02af767e46864c85, store=C 2023-04-19 20:54:39,848 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:54:40,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-19 20:54:40,266 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230419f9359ebd97b44b5a945b6649c61682c4 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419f9359ebd97b44b5a945b6649c61682c4 2023-04-19 20:54:40,671 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=510, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/A/e12fc1c7ccaa4914b03ffbb264eebba6 2023-04-19 20:54:41,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-19 20:54:41,082 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=510 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/B/8757cbd8868142e2a522919c3abfa63b 2023-04-19 20:54:41,492 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=510 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/C/c5859b5a756b477ba4327651d660a1d2 2023-04-19 20:54:41,497 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/A/e12fc1c7ccaa4914b03ffbb264eebba6 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/e12fc1c7ccaa4914b03ffbb264eebba6 2023-04-19 20:54:41,501 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/e12fc1c7ccaa4914b03ffbb264eebba6, entries=150, sequenceid=510, filesize=25.4 K 2023-04-19 20:54:41,501 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/B/8757cbd8868142e2a522919c3abfa63b as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/8757cbd8868142e2a522919c3abfa63b 2023-04-19 20:54:41,505 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/8757cbd8868142e2a522919c3abfa63b, entries=150, sequenceid=510, filesize=11.9 K 2023-04-19 20:54:41,506 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/.tmp/C/c5859b5a756b477ba4327651d660a1d2 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/c5859b5a756b477ba4327651d660a1d2 2023-04-19 20:54:41,509 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/c5859b5a756b477ba4327651d660a1d2, entries=150, sequenceid=510, filesize=11.9 K 2023-04-19 20:54:41,510 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for c7ee8087efe9c3ae02af767e46864c85 in 1663ms, sequenceid=510, compaction requested=true 2023-04-19 20:54:41,530 DEBUG [StoreCloser-TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/03bec7be52ca4b7c9d6044c713757b35, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/101e02bda7a447a5ba7839a7ea2044cf, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/badae060faaa400ea24261b8d7aedfaf, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/eb9caf8a5ede47a4a488c2d07dc15176, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/57bec31be5684a0f92cb8f2d7c0262fb, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/31061a94ffe443c1973cedefddf32e35, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/3c3e2c86178e4d6e9748dcb76dc206ea, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/bb2472d2ab864d8fa9f667fb3aa425f0, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/2be537b2a6874d74af6a8dcc5d336676, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/c2a34e3cf7504fd28de34777ca22d66f, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/0d2ba020b78d4ed78d1adb713eb8d623, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/453272501ae24401b5450a3a6abca5da, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/536796bf8ff64c8aaef8f2c36c3939ca, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/e6ed8f4c953c463d894ef3e50a5c1419, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/1d011e0999b9447392914e99e6e1fbd3, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/29a2064866c94ca38323ebf3efa79cb4, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/44337a04a964422b9470e653acd35d82, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/ecf546bea22e433fbaf955afcea8a3cf, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/70b3efd12fef43cfb355c5ecf42d90d8, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/60939d5542bb409f82114fd3bcdf1811, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/2777891c51d64a84a3e54bfb01fc4a0f, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/f9fa9e3da88c43a6995013c0c70482ed, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/47cbb6a78d5d47ec87b0c66df5b5771f, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/c765a0f5b6ba48998721ef5e2dfc3b7c, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/82bc10ae78dd4c178e79e3cb8d48d793, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/7ba7a588d2df48218eac186d19f16bf8, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/504ee19cfb0c41d79925848025a02888, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/4965da34965e4f8e968be6721650221d] to archive 2023-04-19 20:54:41,532 DEBUG [StoreCloser-TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-19 20:54:41,534 DEBUG [StoreCloser-TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/03bec7be52ca4b7c9d6044c713757b35 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/03bec7be52ca4b7c9d6044c713757b35 2023-04-19 20:54:41,535 DEBUG [StoreCloser-TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/101e02bda7a447a5ba7839a7ea2044cf to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/101e02bda7a447a5ba7839a7ea2044cf 2023-04-19 20:54:41,536 DEBUG [StoreCloser-TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/badae060faaa400ea24261b8d7aedfaf to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/badae060faaa400ea24261b8d7aedfaf 2023-04-19 20:54:41,537 DEBUG [StoreCloser-TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/eb9caf8a5ede47a4a488c2d07dc15176 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/eb9caf8a5ede47a4a488c2d07dc15176 2023-04-19 20:54:41,538 DEBUG [StoreCloser-TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/57bec31be5684a0f92cb8f2d7c0262fb to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/57bec31be5684a0f92cb8f2d7c0262fb 2023-04-19 20:54:41,539 DEBUG [StoreCloser-TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/31061a94ffe443c1973cedefddf32e35 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/31061a94ffe443c1973cedefddf32e35 2023-04-19 20:54:41,540 DEBUG [StoreCloser-TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/3c3e2c86178e4d6e9748dcb76dc206ea to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/3c3e2c86178e4d6e9748dcb76dc206ea 2023-04-19 20:54:41,541 DEBUG [StoreCloser-TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/bb2472d2ab864d8fa9f667fb3aa425f0 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/bb2472d2ab864d8fa9f667fb3aa425f0 2023-04-19 20:54:41,542 DEBUG [StoreCloser-TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/2be537b2a6874d74af6a8dcc5d336676 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/2be537b2a6874d74af6a8dcc5d336676 2023-04-19 20:54:41,543 DEBUG [StoreCloser-TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/c2a34e3cf7504fd28de34777ca22d66f to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/c2a34e3cf7504fd28de34777ca22d66f 2023-04-19 20:54:41,544 DEBUG [StoreCloser-TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/0d2ba020b78d4ed78d1adb713eb8d623 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/0d2ba020b78d4ed78d1adb713eb8d623 2023-04-19 20:54:41,545 DEBUG [StoreCloser-TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/453272501ae24401b5450a3a6abca5da to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/453272501ae24401b5450a3a6abca5da 2023-04-19 20:54:41,546 DEBUG [StoreCloser-TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/536796bf8ff64c8aaef8f2c36c3939ca to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/536796bf8ff64c8aaef8f2c36c3939ca 2023-04-19 20:54:41,547 DEBUG [StoreCloser-TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/e6ed8f4c953c463d894ef3e50a5c1419 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/e6ed8f4c953c463d894ef3e50a5c1419 2023-04-19 20:54:41,548 DEBUG [StoreCloser-TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/1d011e0999b9447392914e99e6e1fbd3 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/1d011e0999b9447392914e99e6e1fbd3 2023-04-19 20:54:41,549 DEBUG [StoreCloser-TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/29a2064866c94ca38323ebf3efa79cb4 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/29a2064866c94ca38323ebf3efa79cb4 2023-04-19 20:54:41,550 DEBUG [StoreCloser-TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/44337a04a964422b9470e653acd35d82 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/44337a04a964422b9470e653acd35d82 2023-04-19 20:54:41,551 DEBUG [StoreCloser-TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/ecf546bea22e433fbaf955afcea8a3cf to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/ecf546bea22e433fbaf955afcea8a3cf 2023-04-19 20:54:41,552 DEBUG [StoreCloser-TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/70b3efd12fef43cfb355c5ecf42d90d8 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/70b3efd12fef43cfb355c5ecf42d90d8 2023-04-19 20:54:41,553 DEBUG [StoreCloser-TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/60939d5542bb409f82114fd3bcdf1811 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/60939d5542bb409f82114fd3bcdf1811 2023-04-19 20:54:41,554 DEBUG [StoreCloser-TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/2777891c51d64a84a3e54bfb01fc4a0f to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/2777891c51d64a84a3e54bfb01fc4a0f 2023-04-19 20:54:41,556 DEBUG [StoreCloser-TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/f9fa9e3da88c43a6995013c0c70482ed to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/f9fa9e3da88c43a6995013c0c70482ed 2023-04-19 20:54:41,557 DEBUG [StoreCloser-TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/47cbb6a78d5d47ec87b0c66df5b5771f to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/47cbb6a78d5d47ec87b0c66df5b5771f 2023-04-19 20:54:41,558 DEBUG [StoreCloser-TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/c765a0f5b6ba48998721ef5e2dfc3b7c to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/c765a0f5b6ba48998721ef5e2dfc3b7c 2023-04-19 20:54:41,559 DEBUG [StoreCloser-TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/82bc10ae78dd4c178e79e3cb8d48d793 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/82bc10ae78dd4c178e79e3cb8d48d793 2023-04-19 20:54:41,560 DEBUG [StoreCloser-TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/7ba7a588d2df48218eac186d19f16bf8 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/7ba7a588d2df48218eac186d19f16bf8 2023-04-19 20:54:41,561 DEBUG [StoreCloser-TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/504ee19cfb0c41d79925848025a02888 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/504ee19cfb0c41d79925848025a02888 2023-04-19 20:54:41,562 DEBUG [StoreCloser-TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/4965da34965e4f8e968be6721650221d to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/4965da34965e4f8e968be6721650221d 2023-04-19 20:54:41,565 DEBUG [StoreCloser-TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/0aba3f28101349eb99fb5dbb80cdfd55, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/9bdaa4f67193473b9d899f5802a4eb47, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/12c4801fea7e4005b2b6d402e1483f57, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/c50116b3f4d64ea8bb8aeb45caeef088, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/009532cfbe874673a4492025c0b126cc, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/6df4a594e7594ceb83c1946f21123132, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/e944bb05f5534a5fb5d3caa0bdea0e1e, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/35655288ccca4a5d9569eaef5c019ea4, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/4d84447e502d46c493e355d443136aa5, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/293e606b965d486394b4cc17d9606e69, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/b2019f9006c749b5b60c5a36b4338c1e, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/efcb2c3696da4e81a524a316e83b9536, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/6d3341a472ae4b2896debd09305e0c4a, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/a19cd3aeeab94e3b829aa79e558ca40f, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/fdc6a3f0271347078b1d3833584608a6, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/c3be8268e328449f998c7138ee775f6b, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/96cda4f49f154fc58d353aa5eac8ee36, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/3fd5c0de13ed46c6ad526a162460340c, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/8f53f8ac6c6c4962a8475ec89300ef67, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/6db9953b384047febc10f2289ae5ff13, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/c8caf13fe1594064bf33ee22afa5d125, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/8aa9018bc5b04eab9765f5b3b73b5244, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/6888c6d185f940c2a9d741405f776996, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/1ba67e80777043219727014b9e100ada, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/f649d570960f47519c41ae06824b0d95, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/1e5b150f784e430b830cf6fb3fa2beb4, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/7b31eeb19d614cd88b6e5700bc4efcba, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/a56cd1e8eb3b413483cd2ad3a5851687] to archive 2023-04-19 20:54:41,566 DEBUG [StoreCloser-TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-19 20:54:41,568 DEBUG [StoreCloser-TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/0aba3f28101349eb99fb5dbb80cdfd55 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/0aba3f28101349eb99fb5dbb80cdfd55 2023-04-19 20:54:41,569 DEBUG [StoreCloser-TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/9bdaa4f67193473b9d899f5802a4eb47 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/9bdaa4f67193473b9d899f5802a4eb47 2023-04-19 20:54:41,570 DEBUG [StoreCloser-TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/12c4801fea7e4005b2b6d402e1483f57 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/12c4801fea7e4005b2b6d402e1483f57 2023-04-19 20:54:41,571 DEBUG [StoreCloser-TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/c50116b3f4d64ea8bb8aeb45caeef088 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/c50116b3f4d64ea8bb8aeb45caeef088 2023-04-19 20:54:41,572 DEBUG [StoreCloser-TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/009532cfbe874673a4492025c0b126cc to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/009532cfbe874673a4492025c0b126cc 2023-04-19 20:54:41,573 DEBUG [StoreCloser-TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/6df4a594e7594ceb83c1946f21123132 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/6df4a594e7594ceb83c1946f21123132 2023-04-19 20:54:41,574 DEBUG [StoreCloser-TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/e944bb05f5534a5fb5d3caa0bdea0e1e to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/e944bb05f5534a5fb5d3caa0bdea0e1e 2023-04-19 20:54:41,575 DEBUG [StoreCloser-TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/35655288ccca4a5d9569eaef5c019ea4 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/35655288ccca4a5d9569eaef5c019ea4 2023-04-19 20:54:41,576 DEBUG [StoreCloser-TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/4d84447e502d46c493e355d443136aa5 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/4d84447e502d46c493e355d443136aa5 2023-04-19 20:54:41,577 DEBUG [StoreCloser-TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/293e606b965d486394b4cc17d9606e69 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/293e606b965d486394b4cc17d9606e69 2023-04-19 20:54:41,578 DEBUG [StoreCloser-TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/b2019f9006c749b5b60c5a36b4338c1e to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/b2019f9006c749b5b60c5a36b4338c1e 2023-04-19 20:54:41,579 DEBUG [StoreCloser-TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/efcb2c3696da4e81a524a316e83b9536 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/efcb2c3696da4e81a524a316e83b9536 2023-04-19 20:54:41,580 DEBUG [StoreCloser-TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/6d3341a472ae4b2896debd09305e0c4a to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/6d3341a472ae4b2896debd09305e0c4a 2023-04-19 20:54:41,581 DEBUG [StoreCloser-TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/a19cd3aeeab94e3b829aa79e558ca40f to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/a19cd3aeeab94e3b829aa79e558ca40f 2023-04-19 20:54:41,583 DEBUG [StoreCloser-TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/fdc6a3f0271347078b1d3833584608a6 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/fdc6a3f0271347078b1d3833584608a6 2023-04-19 20:54:41,584 DEBUG [StoreCloser-TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/c3be8268e328449f998c7138ee775f6b to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/c3be8268e328449f998c7138ee775f6b 2023-04-19 20:54:41,585 DEBUG [StoreCloser-TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/96cda4f49f154fc58d353aa5eac8ee36 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/96cda4f49f154fc58d353aa5eac8ee36 2023-04-19 20:54:41,586 DEBUG [StoreCloser-TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/3fd5c0de13ed46c6ad526a162460340c to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/3fd5c0de13ed46c6ad526a162460340c 2023-04-19 20:54:41,587 DEBUG [StoreCloser-TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/8f53f8ac6c6c4962a8475ec89300ef67 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/8f53f8ac6c6c4962a8475ec89300ef67 2023-04-19 20:54:41,588 DEBUG [StoreCloser-TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/6db9953b384047febc10f2289ae5ff13 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/6db9953b384047febc10f2289ae5ff13 2023-04-19 20:54:41,589 DEBUG [StoreCloser-TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/c8caf13fe1594064bf33ee22afa5d125 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/c8caf13fe1594064bf33ee22afa5d125 2023-04-19 20:54:41,590 DEBUG [StoreCloser-TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/8aa9018bc5b04eab9765f5b3b73b5244 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/8aa9018bc5b04eab9765f5b3b73b5244 2023-04-19 20:54:41,592 DEBUG [StoreCloser-TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/6888c6d185f940c2a9d741405f776996 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/6888c6d185f940c2a9d741405f776996 2023-04-19 20:54:41,593 DEBUG [StoreCloser-TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/1ba67e80777043219727014b9e100ada to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/1ba67e80777043219727014b9e100ada 2023-04-19 20:54:41,594 DEBUG [StoreCloser-TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/f649d570960f47519c41ae06824b0d95 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/f649d570960f47519c41ae06824b0d95 2023-04-19 20:54:41,595 DEBUG [StoreCloser-TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/1e5b150f784e430b830cf6fb3fa2beb4 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/1e5b150f784e430b830cf6fb3fa2beb4 2023-04-19 20:54:41,596 DEBUG [StoreCloser-TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/7b31eeb19d614cd88b6e5700bc4efcba to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/7b31eeb19d614cd88b6e5700bc4efcba 2023-04-19 20:54:41,597 DEBUG [StoreCloser-TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/a56cd1e8eb3b413483cd2ad3a5851687 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/a56cd1e8eb3b413483cd2ad3a5851687 2023-04-19 20:54:41,600 DEBUG [StoreCloser-TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/7334fe85469e4371908bd37f2ae522e1, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/69aa733f9b70435ab3b80dfd86dcdbba, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/7e79e56e618c487f959113e68f83d37d, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/639d44ae22504a589807fcadc1e8e230, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/e01e4a7c480242428ae5667c87c4b5d7, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/b963dd73018b49d78c8e3816f6e38fd1, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/3ad89c0678f84aed886a6a97bf1d56c6, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/6706bc3aa4054b93b3a4d6dec1537c3f, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/73b24293d2364391b6a486932f8b0a30, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/0392f2d555904ac6a78d7a71643261ab, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/82ed735294aa4455b95553ef97020bab, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/dbcb253d0c6446ac9d7b512ae1f30263, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/469dabb7d280474eadb90a7abfc38076, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/9aa7e84de7f24bd5a24e41acbc9f3709, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/14718df5e08f41daa6027bbc6b8a1339, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/62ef60b328604a43ba048fec23bf9792, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/4217aa3d5153432cb855379e8c506418, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/d1f1b9052223440fa3ccec51bd74a9f8, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/3250fa01f6fd4038bf1f73b7e7e4f2cb, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/f0c33174d9684abda8f058a9efbe43fc, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/954c3406cbda45a4b016a86d6f1c7b8c, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/873cb6ed3c2245e7a75fc47062026599, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/afb382a30225400fafa01dc2ed137da9, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/183283551a36445aa0a69020785d8166, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/27f688ef68364a819ae0f0ffce6000f3, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/0a034a27700d487ca3473d6de31eb19f, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/c60fbab40ecd4f949b59e1b97c06505a, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/c44065b7e2204744b671616cdcfdf04d] to archive 2023-04-19 20:54:41,601 DEBUG [StoreCloser-TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-19 20:54:41,602 DEBUG [StoreCloser-TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/7334fe85469e4371908bd37f2ae522e1 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/7334fe85469e4371908bd37f2ae522e1 2023-04-19 20:54:41,604 DEBUG [StoreCloser-TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/69aa733f9b70435ab3b80dfd86dcdbba to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/69aa733f9b70435ab3b80dfd86dcdbba 2023-04-19 20:54:41,605 DEBUG [StoreCloser-TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/7e79e56e618c487f959113e68f83d37d to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/7e79e56e618c487f959113e68f83d37d 2023-04-19 20:54:41,606 DEBUG [StoreCloser-TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/639d44ae22504a589807fcadc1e8e230 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/639d44ae22504a589807fcadc1e8e230 2023-04-19 20:54:41,607 DEBUG [StoreCloser-TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/e01e4a7c480242428ae5667c87c4b5d7 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/e01e4a7c480242428ae5667c87c4b5d7 2023-04-19 20:54:41,609 DEBUG [StoreCloser-TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/b963dd73018b49d78c8e3816f6e38fd1 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/b963dd73018b49d78c8e3816f6e38fd1 2023-04-19 20:54:41,610 DEBUG [StoreCloser-TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/3ad89c0678f84aed886a6a97bf1d56c6 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/3ad89c0678f84aed886a6a97bf1d56c6 2023-04-19 20:54:41,611 DEBUG [StoreCloser-TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/6706bc3aa4054b93b3a4d6dec1537c3f to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/6706bc3aa4054b93b3a4d6dec1537c3f 2023-04-19 20:54:41,612 DEBUG [StoreCloser-TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/73b24293d2364391b6a486932f8b0a30 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/73b24293d2364391b6a486932f8b0a30 2023-04-19 20:54:41,613 DEBUG [StoreCloser-TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/0392f2d555904ac6a78d7a71643261ab to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/0392f2d555904ac6a78d7a71643261ab 2023-04-19 20:54:41,615 DEBUG [StoreCloser-TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/82ed735294aa4455b95553ef97020bab to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/82ed735294aa4455b95553ef97020bab 2023-04-19 20:54:41,616 DEBUG [StoreCloser-TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/dbcb253d0c6446ac9d7b512ae1f30263 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/dbcb253d0c6446ac9d7b512ae1f30263 2023-04-19 20:54:41,617 DEBUG [StoreCloser-TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/469dabb7d280474eadb90a7abfc38076 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/469dabb7d280474eadb90a7abfc38076 2023-04-19 20:54:41,618 DEBUG [StoreCloser-TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/9aa7e84de7f24bd5a24e41acbc9f3709 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/9aa7e84de7f24bd5a24e41acbc9f3709 2023-04-19 20:54:41,619 DEBUG [StoreCloser-TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/14718df5e08f41daa6027bbc6b8a1339 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/14718df5e08f41daa6027bbc6b8a1339 2023-04-19 20:54:41,621 DEBUG [StoreCloser-TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/62ef60b328604a43ba048fec23bf9792 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/62ef60b328604a43ba048fec23bf9792 2023-04-19 20:54:41,622 DEBUG [StoreCloser-TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/4217aa3d5153432cb855379e8c506418 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/4217aa3d5153432cb855379e8c506418 2023-04-19 20:54:41,623 DEBUG [StoreCloser-TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/d1f1b9052223440fa3ccec51bd74a9f8 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/d1f1b9052223440fa3ccec51bd74a9f8 2023-04-19 20:54:41,624 DEBUG [StoreCloser-TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/3250fa01f6fd4038bf1f73b7e7e4f2cb to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/3250fa01f6fd4038bf1f73b7e7e4f2cb 2023-04-19 20:54:41,626 DEBUG [StoreCloser-TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/f0c33174d9684abda8f058a9efbe43fc to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/f0c33174d9684abda8f058a9efbe43fc 2023-04-19 20:54:41,627 DEBUG [StoreCloser-TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/954c3406cbda45a4b016a86d6f1c7b8c to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/954c3406cbda45a4b016a86d6f1c7b8c 2023-04-19 20:54:41,629 DEBUG [StoreCloser-TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/873cb6ed3c2245e7a75fc47062026599 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/873cb6ed3c2245e7a75fc47062026599 2023-04-19 20:54:41,630 DEBUG [StoreCloser-TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/afb382a30225400fafa01dc2ed137da9 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/afb382a30225400fafa01dc2ed137da9 2023-04-19 20:54:41,632 DEBUG [StoreCloser-TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/183283551a36445aa0a69020785d8166 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/183283551a36445aa0a69020785d8166 2023-04-19 20:54:41,633 DEBUG [StoreCloser-TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/27f688ef68364a819ae0f0ffce6000f3 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/27f688ef68364a819ae0f0ffce6000f3 2023-04-19 20:54:41,634 DEBUG [StoreCloser-TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/0a034a27700d487ca3473d6de31eb19f to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/0a034a27700d487ca3473d6de31eb19f 2023-04-19 20:54:41,635 DEBUG [StoreCloser-TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/c60fbab40ecd4f949b59e1b97c06505a to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/c60fbab40ecd4f949b59e1b97c06505a 2023-04-19 20:54:41,636 DEBUG [StoreCloser-TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/c44065b7e2204744b671616cdcfdf04d to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/c44065b7e2204744b671616cdcfdf04d 2023-04-19 20:54:41,641 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/recovered.edits/513.seqid, newMaxSeqId=513, maxSeqId=4 2023-04-19 20:54:41,642 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85. 2023-04-19 20:54:41,643 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for c7ee8087efe9c3ae02af767e46864c85: 2023-04-19 20:54:41,644 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed c7ee8087efe9c3ae02af767e46864c85 2023-04-19 20:54:41,644 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=c7ee8087efe9c3ae02af767e46864c85, regionState=CLOSED 2023-04-19 20:54:41,645 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681937681644"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681937681644"}]},"ts":"1681937681644"} 2023-04-19 20:54:41,647 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=26, resume processing ppid=25 2023-04-19 20:54:41,647 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=26, ppid=25, state=SUCCESS; CloseRegionProcedure c7ee8087efe9c3ae02af767e46864c85, server=jenkins-hbase4.apache.org,35247,1681937616971 in 2.6770 sec 2023-04-19 20:54:41,649 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=25, resume processing ppid=24 2023-04-19 20:54:41,649 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=25, ppid=24, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=c7ee8087efe9c3ae02af767e46864c85, UNASSIGN in 2.6810 sec 2023-04-19 20:54:41,649 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681937681649"}]},"ts":"1681937681649"} 2023-04-19 20:54:41,650 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-19 20:54:41,651 INFO [PEWorker-5] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-19 20:54:41,653 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=24, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 2.6950 sec 2023-04-19 20:54:43,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-19 20:54:43,066 INFO [Listener at localhost/34893] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 24 completed 2023-04-19 20:54:43,067 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-19 20:54:43,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure2.ProcedureExecutor(1029): Stored pid=27, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-19 20:54:43,069 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=27, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-19 20:54:43,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1227): Checking to see if procedure is done pid=27 2023-04-19 20:54:43,070 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=27, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-19 20:54:43,075 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/.tmp/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85 2023-04-19 20:54:43,077 DEBUG [HFileArchiver-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/.tmp/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A, FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/.tmp/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B, FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/.tmp/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C, FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/.tmp/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/recovered.edits] 2023-04-19 20:54:43,080 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/.tmp/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/c03bbc3cb0c54098b1d572145bb5bb37 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/c03bbc3cb0c54098b1d572145bb5bb37 2023-04-19 20:54:43,081 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/.tmp/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/c877773aa7ec4583ba2401ea4847c9bd to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/c877773aa7ec4583ba2401ea4847c9bd 2023-04-19 20:54:43,083 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/.tmp/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/e12fc1c7ccaa4914b03ffbb264eebba6 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/A/e12fc1c7ccaa4914b03ffbb264eebba6 2023-04-19 20:54:43,085 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/.tmp/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/36296c4afdc04b0297d49b30ee908166 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/36296c4afdc04b0297d49b30ee908166 2023-04-19 20:54:43,086 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/.tmp/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/3cbfbf56d662491f8345cc371ed23e56 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/3cbfbf56d662491f8345cc371ed23e56 2023-04-19 20:54:43,087 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/.tmp/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/8757cbd8868142e2a522919c3abfa63b to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/B/8757cbd8868142e2a522919c3abfa63b 2023-04-19 20:54:43,089 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/.tmp/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/258956a6880843759da6e10706e85634 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/258956a6880843759da6e10706e85634 2023-04-19 20:54:43,090 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/.tmp/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/289896eb506f4d01b21fa5dfc8181351 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/289896eb506f4d01b21fa5dfc8181351 2023-04-19 20:54:43,091 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/.tmp/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/c5859b5a756b477ba4327651d660a1d2 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/C/c5859b5a756b477ba4327651d660a1d2 2023-04-19 20:54:43,094 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/.tmp/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/recovered.edits/513.seqid to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85/recovered.edits/513.seqid 2023-04-19 20:54:43,094 DEBUG [HFileArchiver-5] backup.HFileArchiver(596): Deleted hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/.tmp/data/default/TestAcidGuarantees/c7ee8087efe9c3ae02af767e46864c85 2023-04-19 20:54:43,094 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-19 20:54:43,095 DEBUG [PEWorker-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-19 20:54:43,095 DEBUG [PEWorker-1] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-04-19 20:54:43,099 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419087340cf731f4534b45c4cfaad10fb96 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419087340cf731f4534b45c4cfaad10fb96 2023-04-19 20:54:43,100 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304190e95a73ac6f74ef6aa9bfe47c87e113a to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304190e95a73ac6f74ef6aa9bfe47c87e113a 2023-04-19 20:54:43,101 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304191874794a078f4516921021818046cd2c to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304191874794a078f4516921021818046cd2c 2023-04-19 20:54:43,102 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304191c6ec0b91acd4661ada72f76bfe5c2d2 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304191c6ec0b91acd4661ada72f76bfe5c2d2 2023-04-19 20:54:43,103 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041925466cd5005a44578e9be0c7402867c6 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041925466cd5005a44578e9be0c7402867c6 2023-04-19 20:54:43,104 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304192fdf6f395a604d09b95e7ac1fba8df12 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304192fdf6f395a604d09b95e7ac1fba8df12 2023-04-19 20:54:43,105 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419362b4d4299874edab5306ffb9454ca1b to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419362b4d4299874edab5306ffb9454ca1b 2023-04-19 20:54:43,106 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304193a9c6352d363476283bc46dfcd3549c8 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304193a9c6352d363476283bc46dfcd3549c8 2023-04-19 20:54:43,107 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304194e38d567d6e84913b2328030664a1953 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304194e38d567d6e84913b2328030664a1953 2023-04-19 20:54:43,108 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041954a4732e23d84f3484730dbfe675e139 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041954a4732e23d84f3484730dbfe675e139 2023-04-19 20:54:43,109 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419595b72847c864ff2a1fad3fbd031f50f to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419595b72847c864ff2a1fad3fbd031f50f 2023-04-19 20:54:43,110 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419721296777b43468ab8a16455e46e1cf5 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419721296777b43468ab8a16455e46e1cf5 2023-04-19 20:54:43,111 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304197d217cc663c549a9bfa900cd7ab03e4a to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304197d217cc663c549a9bfa900cd7ab03e4a 2023-04-19 20:54:43,112 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304197f8ec5398e17499d9da3fed46b2afdd0 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304197f8ec5398e17499d9da3fed46b2afdd0 2023-04-19 20:54:43,113 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304198cc55b9bc9af4fa1b34b9d25f023aee2 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304198cc55b9bc9af4fa1b34b9d25f023aee2 2023-04-19 20:54:43,114 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304199166b8188ce442d49557ff1369e59dbb to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304199166b8188ce442d49557ff1369e59dbb 2023-04-19 20:54:43,115 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419aea8563ede744918b8e2f63546d53800 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419aea8563ede744918b8e2f63546d53800 2023-04-19 20:54:43,116 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419b4b9707441344bbd99a53beae40ab597 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419b4b9707441344bbd99a53beae40ab597 2023-04-19 20:54:43,117 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419bab3c6cfe3a848bcb0bc4ed0139e5a4f to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419bab3c6cfe3a848bcb0bc4ed0139e5a4f 2023-04-19 20:54:43,118 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419c8a3ff2f4223499c871ca8284589e75f to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419c8a3ff2f4223499c871ca8284589e75f 2023-04-19 20:54:43,119 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419e70b71edc0d64773902ac5340d8231d4 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419e70b71edc0d64773902ac5340d8231d4 2023-04-19 20:54:43,120 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419ea2b1b564b54469abc8ce6fd48077134 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419ea2b1b564b54469abc8ce6fd48077134 2023-04-19 20:54:43,121 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419ec58f7bd9f0c4fc988f2c37cf3cbbc3f to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419ec58f7bd9f0c4fc988f2c37cf3cbbc3f 2023-04-19 20:54:43,122 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419ee7607845217489fa0be7a16c4f9deb9 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419ee7607845217489fa0be7a16c4f9deb9 2023-04-19 20:54:43,124 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419f9359ebd97b44b5a945b6649c61682c4 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419f9359ebd97b44b5a945b6649c61682c4 2023-04-19 20:54:43,124 DEBUG [PEWorker-1] backup.HFileArchiver(596): Deleted hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-19 20:54:43,126 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=27, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-19 20:54:43,128 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-19 20:54:43,130 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-19 20:54:43,131 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=27, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-19 20:54:43,131 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-19 20:54:43,131 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1681937683131"}]},"ts":"9223372036854775807"} 2023-04-19 20:54:43,133 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-19 20:54:43,133 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => c7ee8087efe9c3ae02af767e46864c85, NAME => 'TestAcidGuarantees,,1681937653963.c7ee8087efe9c3ae02af767e46864c85.', STARTKEY => '', ENDKEY => ''}] 2023-04-19 20:54:43,133 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-19 20:54:43,133 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1681937683133"}]},"ts":"9223372036854775807"} 2023-04-19 20:54:43,134 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-19 20:54:43,136 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=27, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-19 20:54:43,137 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=27, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 69 msec 2023-04-19 20:54:43,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1227): Checking to see if procedure is done pid=27 2023-04-19 20:54:43,171 INFO [Listener at localhost/34893] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 27 completed 2023-04-19 20:54:43,183 INFO [Listener at localhost/34893] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobMixedAtomicity Thread=227 (was 230), OpenFileDescriptor=583 (was 545) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=348 (was 396), ProcessCount=169 (was 169), AvailableMemoryMB=5561 (was 5724) 2023-04-19 20:54:43,194 INFO [Listener at localhost/34893] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testGetAtomicity Thread=227, OpenFileDescriptor=583, MaxFileDescriptor=60000, SystemLoadAverage=348, ProcessCount=169, AvailableMemoryMB=5561 2023-04-19 20:54:43,196 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-19 20:54:43,196 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-19 20:54:43,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure2.ProcedureExecutor(1029): Stored pid=28, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-19 20:54:43,198 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-19 20:54:43,198 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 28 2023-04-19 20:54:43,199 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-19 20:54:43,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-19 20:54:43,200 DEBUG [HFileArchiver-6] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/.tmp/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4 2023-04-19 20:54:43,200 DEBUG [HFileArchiver-6] backup.HFileArchiver(153): Directory hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/.tmp/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4 empty. 2023-04-19 20:54:43,201 DEBUG [HFileArchiver-6] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/.tmp/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4 2023-04-19 20:54:43,201 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-19 20:54:43,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-19 20:54:43,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-19 20:54:43,609 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-19 20:54:43,611 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 09bc544118dfd67701ac39d6da1a53f4, NAME => 'TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/.tmp 2023-04-19 20:54:43,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-19 20:54:44,017 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-19 20:54:44,017 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 09bc544118dfd67701ac39d6da1a53f4, disabling compactions & flushes 2023-04-19 20:54:44,017 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. 2023-04-19 20:54:44,017 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. 2023-04-19 20:54:44,017 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. after waiting 0 ms 2023-04-19 20:54:44,017 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. 2023-04-19 20:54:44,017 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. 2023-04-19 20:54:44,017 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 09bc544118dfd67701ac39d6da1a53f4: 2023-04-19 20:54:44,019 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-19 20:54:44,020 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681937684020"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681937684020"}]},"ts":"1681937684020"} 2023-04-19 20:54:44,021 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-19 20:54:44,022 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-19 20:54:44,022 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681937684022"}]},"ts":"1681937684022"} 2023-04-19 20:54:44,023 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-19 20:54:44,025 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=09bc544118dfd67701ac39d6da1a53f4, ASSIGN}] 2023-04-19 20:54:44,027 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=09bc544118dfd67701ac39d6da1a53f4, ASSIGN 2023-04-19 20:54:44,027 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=09bc544118dfd67701ac39d6da1a53f4, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,35247,1681937616971; forceNewPlan=false, retain=false 2023-04-19 20:54:44,178 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=09bc544118dfd67701ac39d6da1a53f4, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:44,179 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681937684178"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681937684178"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681937684178"}]},"ts":"1681937684178"} 2023-04-19 20:54:44,180 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=30, ppid=29, state=RUNNABLE; OpenRegionProcedure 09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971}] 2023-04-19 20:54:44,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-19 20:54:44,335 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. 2023-04-19 20:54:44,335 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 09bc544118dfd67701ac39d6da1a53f4, NAME => 'TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4.', STARTKEY => '', ENDKEY => ''} 2023-04-19 20:54:44,335 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 09bc544118dfd67701ac39d6da1a53f4 2023-04-19 20:54:44,335 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-19 20:54:44,335 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 09bc544118dfd67701ac39d6da1a53f4 2023-04-19 20:54:44,335 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 09bc544118dfd67701ac39d6da1a53f4 2023-04-19 20:54:44,337 INFO [StoreOpener-09bc544118dfd67701ac39d6da1a53f4-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 09bc544118dfd67701ac39d6da1a53f4 2023-04-19 20:54:44,339 DEBUG [StoreOpener-09bc544118dfd67701ac39d6da1a53f4-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A 2023-04-19 20:54:44,339 DEBUG [StoreOpener-09bc544118dfd67701ac39d6da1a53f4-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A 2023-04-19 20:54:44,339 INFO [StoreOpener-09bc544118dfd67701ac39d6da1a53f4-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-19 20:54:44,339 INFO [StoreOpener-09bc544118dfd67701ac39d6da1a53f4-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 09bc544118dfd67701ac39d6da1a53f4 columnFamilyName A 2023-04-19 20:54:44,340 INFO [StoreOpener-09bc544118dfd67701ac39d6da1a53f4-1] regionserver.HStore(310): Store=09bc544118dfd67701ac39d6da1a53f4/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-19 20:54:44,340 INFO [StoreOpener-09bc544118dfd67701ac39d6da1a53f4-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 09bc544118dfd67701ac39d6da1a53f4 2023-04-19 20:54:44,341 DEBUG [StoreOpener-09bc544118dfd67701ac39d6da1a53f4-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B 2023-04-19 20:54:44,341 DEBUG [StoreOpener-09bc544118dfd67701ac39d6da1a53f4-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B 2023-04-19 20:54:44,341 INFO [StoreOpener-09bc544118dfd67701ac39d6da1a53f4-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-19 20:54:44,341 INFO [StoreOpener-09bc544118dfd67701ac39d6da1a53f4-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 09bc544118dfd67701ac39d6da1a53f4 columnFamilyName B 2023-04-19 20:54:44,342 INFO [StoreOpener-09bc544118dfd67701ac39d6da1a53f4-1] regionserver.HStore(310): Store=09bc544118dfd67701ac39d6da1a53f4/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-19 20:54:44,342 INFO [StoreOpener-09bc544118dfd67701ac39d6da1a53f4-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 09bc544118dfd67701ac39d6da1a53f4 2023-04-19 20:54:44,343 DEBUG [StoreOpener-09bc544118dfd67701ac39d6da1a53f4-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C 2023-04-19 20:54:44,343 DEBUG [StoreOpener-09bc544118dfd67701ac39d6da1a53f4-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C 2023-04-19 20:54:44,343 INFO [StoreOpener-09bc544118dfd67701ac39d6da1a53f4-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-19 20:54:44,344 INFO [StoreOpener-09bc544118dfd67701ac39d6da1a53f4-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 09bc544118dfd67701ac39d6da1a53f4 columnFamilyName C 2023-04-19 20:54:44,344 INFO [StoreOpener-09bc544118dfd67701ac39d6da1a53f4-1] regionserver.HStore(310): Store=09bc544118dfd67701ac39d6da1a53f4/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-19 20:54:44,344 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. 2023-04-19 20:54:44,345 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4 2023-04-19 20:54:44,345 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4 2023-04-19 20:54:44,347 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-19 20:54:44,348 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 09bc544118dfd67701ac39d6da1a53f4 2023-04-19 20:54:44,350 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-19 20:54:44,350 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 09bc544118dfd67701ac39d6da1a53f4; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64664288, jitterRate=-0.03642702102661133}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-19 20:54:44,350 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 09bc544118dfd67701ac39d6da1a53f4: 2023-04-19 20:54:44,351 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4., pid=30, masterSystemTime=1681937684331 2023-04-19 20:54:44,352 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. 2023-04-19 20:54:44,352 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. 2023-04-19 20:54:44,353 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=09bc544118dfd67701ac39d6da1a53f4, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:44,353 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681937684353"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681937684353"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681937684353"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681937684353"}]},"ts":"1681937684353"} 2023-04-19 20:54:44,355 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=30, resume processing ppid=29 2023-04-19 20:54:44,355 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=30, ppid=29, state=SUCCESS; OpenRegionProcedure 09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 in 174 msec 2023-04-19 20:54:44,357 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=29, resume processing ppid=28 2023-04-19 20:54:44,357 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=29, ppid=28, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=09bc544118dfd67701ac39d6da1a53f4, ASSIGN in 330 msec 2023-04-19 20:54:44,357 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-19 20:54:44,357 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681937684357"}]},"ts":"1681937684357"} 2023-04-19 20:54:44,358 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-19 20:54:44,361 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-19 20:54:44,362 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=28, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1650 sec 2023-04-19 20:54:45,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-19 20:54:45,304 INFO [Listener at localhost/34893] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 28 completed 2023-04-19 20:54:45,305 DEBUG [Listener at localhost/34893] zookeeper.ReadOnlyZKClient(139): Connect 0x53921272 to 127.0.0.1:58736 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-19 20:54:45,310 DEBUG [Listener at localhost/34893] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2d7cdc2b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-19 20:54:45,311 DEBUG [Listener at localhost/34893] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-19 20:54:45,312 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:34498, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-19 20:54:45,313 DEBUG [Listener at localhost/34893] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-19 20:54:45,315 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:45942, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-19 20:54:45,316 DEBUG [Listener at localhost/34893] zookeeper.ReadOnlyZKClient(139): Connect 0x13544701 to 127.0.0.1:58736 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-19 20:54:45,319 DEBUG [Listener at localhost/34893] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5096c69b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-19 20:54:45,320 DEBUG [Listener at localhost/34893] zookeeper.ReadOnlyZKClient(139): Connect 0x10b79c2c to 127.0.0.1:58736 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-19 20:54:45,323 DEBUG [Listener at localhost/34893] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@216aad74, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-19 20:54:45,324 DEBUG [Listener at localhost/34893] zookeeper.ReadOnlyZKClient(139): Connect 0x5a5a81fa to 127.0.0.1:58736 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-19 20:54:45,331 DEBUG [Listener at localhost/34893] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@76203061, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-19 20:54:45,331 DEBUG [Listener at localhost/34893] zookeeper.ReadOnlyZKClient(139): Connect 0x3562c7bf to 127.0.0.1:58736 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-19 20:54:45,334 DEBUG [Listener at localhost/34893] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3b39d5b8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-19 20:54:45,335 DEBUG [Listener at localhost/34893] zookeeper.ReadOnlyZKClient(139): Connect 0x6705d115 to 127.0.0.1:58736 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-19 20:54:45,338 DEBUG [Listener at localhost/34893] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@44f9bd75, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-19 20:54:45,338 DEBUG [Listener at localhost/34893] zookeeper.ReadOnlyZKClient(139): Connect 0x5e5b1c14 to 127.0.0.1:58736 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-19 20:54:45,342 DEBUG [Listener at localhost/34893] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@574000bf, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-19 20:54:45,342 DEBUG [Listener at localhost/34893] zookeeper.ReadOnlyZKClient(139): Connect 0x4f29c1c4 to 127.0.0.1:58736 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-19 20:54:45,347 DEBUG [Listener at localhost/34893] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4994df89, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-19 20:54:45,347 DEBUG [Listener at localhost/34893] zookeeper.ReadOnlyZKClient(139): Connect 0x681ace6c to 127.0.0.1:58736 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-19 20:54:45,353 DEBUG [Listener at localhost/34893] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@69bc154a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-19 20:54:45,354 DEBUG [Listener at localhost/34893] zookeeper.ReadOnlyZKClient(139): Connect 0x2b4fa5d3 to 127.0.0.1:58736 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-19 20:54:45,358 DEBUG [Listener at localhost/34893] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@15186778, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-19 20:54:45,358 DEBUG [Listener at localhost/34893] zookeeper.ReadOnlyZKClient(139): Connect 0x45de5ad8 to 127.0.0.1:58736 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-19 20:54:45,362 DEBUG [Listener at localhost/34893] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@13e90e8d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-19 20:54:45,366 DEBUG [hconnection-0x7c78163c-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-19 20:54:45,367 DEBUG [hconnection-0x52ebddbf-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-19 20:54:45,367 DEBUG [hconnection-0x423608be-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-19 20:54:45,367 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:54:45,369 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:34522, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-19 20:54:45,369 DEBUG [hconnection-0x5dedb68-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-19 20:54:45,370 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:34528, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-19 20:54:45,370 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:34510, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-19 20:54:45,372 DEBUG [hconnection-0x472e01b2-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-19 20:54:45,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:54:45,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:54:45,373 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:54:45,373 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:45,373 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:54:45,373 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:54:45,374 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:45,374 DEBUG [hconnection-0xf85e176-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-19 20:54:45,375 DEBUG [hconnection-0x2b70e3e6-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-19 20:54:45,376 DEBUG [hconnection-0x7ca19371-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-19 20:54:45,375 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:34536, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-19 20:54:45,375 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:34532, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-19 20:54:45,378 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:34540, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-19 20:54:45,375 DEBUG [hconnection-0x4cd46def-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-19 20:54:45,375 DEBUG [hconnection-0x510aa473-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-19 20:54:45,374 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:45,381 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:34552, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-19 20:54:45,381 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:34562, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-19 20:54:45,383 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:45,384 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:45,384 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:34548, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-19 20:54:45,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:45,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:45,384 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:45,384 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:34572, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-19 20:54:45,384 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:54:45,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:45,391 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:45,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:54:45,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:45,391 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:54:45,400 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:54:45,400 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:54:45,400 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:54:45,400 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:45,400 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:54:45,401 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:54:45,401 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:54:45,401 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. 2023-04-19 20:54:45,401 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. started... 2023-04-19 20:54:45,401 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 09bc544118dfd67701ac39d6da1a53f4 3/3 column families, dataSize=20.13 KB heapSize=60.52 KB 2023-04-19 20:54:45,402 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09bc544118dfd67701ac39d6da1a53f4, store=A 2023-04-19 20:54:45,402 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:54:45,402 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09bc544118dfd67701ac39d6da1a53f4, store=B 2023-04-19 20:54:45,402 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:54:45,402 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09bc544118dfd67701ac39d6da1a53f4, store=C 2023-04-19 20:54:45,402 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:54:45,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(9158): Flush requested on 09bc544118dfd67701ac39d6da1a53f4 2023-04-19 20:54:45,404 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. as already flushing 2023-04-19 20:54:45,428 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681937685423, max timestamp=1681937685425Num uniques 859; 2023-04-19 20:54:45,428 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681937685423, max timestamp=1681937685425Num uniques 859; 2023-04-19 20:54:45,428 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681937685423, max timestamp=1681937685425Num uniques 859; 2023-04-19 20:54:45,428 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 20:54:45,428 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 20:54:45,428 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 20:54:45,443 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:45,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34572 deadline: 1681937745440, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:45,444 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:45,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34562 deadline: 1681937745440, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:45,445 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:45,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34548 deadline: 1681937745441, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:45,445 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1681937685436, max timestamp=1681937685442Num uniques 1018; 2023-04-19 20:54:45,445 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-19 20:54:45,446 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1681937685436, max timestamp=1681937685442Num uniques 1018; 2023-04-19 20:54:45,446 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-19 20:54:45,448 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1681937685436, max timestamp=1681937685442Num uniques 1018; 2023-04-19 20:54:45,448 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-19 20:54:45,483 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:45,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34540 deadline: 1681937745481, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:45,486 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:45,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34552 deadline: 1681937745483, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:45,546 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:45,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34572 deadline: 1681937745545, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:45,547 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:45,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34562 deadline: 1681937745545, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:45,548 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:45,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34548 deadline: 1681937745546, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:45,587 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:45,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34540 deadline: 1681937745584, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:45,589 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:45,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34552 deadline: 1681937745587, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:45,749 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:45,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34572 deadline: 1681937745749, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:45,750 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:45,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34562 deadline: 1681937745749, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:45,752 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:45,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34548 deadline: 1681937745750, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:45,790 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:45,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34540 deadline: 1681937745788, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:45,791 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:45,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34552 deadline: 1681937745790, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:45,816 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=10 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/A/f6063291209d427bad7d0a0922ab5ac4 2023-04-19 20:54:46,053 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:46,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34572 deadline: 1681937746052, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:46,053 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:46,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34562 deadline: 1681937746052, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:46,054 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:46,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34548 deadline: 1681937746053, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:46,092 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:46,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34540 deadline: 1681937746091, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:46,095 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:46,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34552 deadline: 1681937746094, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:46,228 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=10 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/B/82f6f8fd358241138162f76c87217e99 2023-04-19 20:54:46,245 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=10 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/C/e8344ec11af845dc877d43c20f89f903 2023-04-19 20:54:46,249 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/A/f6063291209d427bad7d0a0922ab5ac4 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/f6063291209d427bad7d0a0922ab5ac4 2023-04-19 20:54:46,254 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/f6063291209d427bad7d0a0922ab5ac4, entries=150, sequenceid=10, filesize=11.6 K 2023-04-19 20:54:46,254 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/B/82f6f8fd358241138162f76c87217e99 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/82f6f8fd358241138162f76c87217e99 2023-04-19 20:54:46,261 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/82f6f8fd358241138162f76c87217e99, entries=150, sequenceid=10, filesize=11.6 K 2023-04-19 20:54:46,262 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/C/e8344ec11af845dc877d43c20f89f903 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/e8344ec11af845dc877d43c20f89f903 2023-04-19 20:54:46,265 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/e8344ec11af845dc877d43c20f89f903, entries=150, sequenceid=10, filesize=11.6 K 2023-04-19 20:54:46,266 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=140.89 KB/144270 for 09bc544118dfd67701ac39d6da1a53f4 in 865ms, sequenceid=10, compaction requested=false 2023-04-19 20:54:46,266 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 09bc544118dfd67701ac39d6da1a53f4: 2023-04-19 20:54:46,266 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. 2023-04-19 20:54:46,266 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:54:46,266 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:54:46,266 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:46,266 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:54:46,267 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:54:46,269 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:46,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:46,269 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:46,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:46,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:46,269 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:46,269 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:54:46,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:46,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:46,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:46,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:46,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:46,271 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:54:46,271 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ee87801[Count = 0] remaining members to acquire global barrier 2023-04-19 20:54:46,271 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:54:46,271 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:46,272 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:46,272 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:46,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:46,272 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:54:46,272 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:54:46,272 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:54:46,272 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:46,272 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:54:46,274 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:54:46,274 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:46,274 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:46,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:46,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:46,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:46,274 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:54:46,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:46,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:46,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:46,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:46,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:46,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:46,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:46,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:54:46,276 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:54:46,276 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:54:46,276 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:54:46,276 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:54:46,276 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:54:46,277 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:54:46,278 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:46,278 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:46,278 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:46,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:46,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:46,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:46,278 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:46,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:46,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:46,279 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:46,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:46,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:46,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:46,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:46,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:46,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:46,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:46,280 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:46,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:46,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:46,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:46,283 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:46,283 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:46,283 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:46,283 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:54:46,283 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:46,283 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:46,284 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:46,284 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:46,284 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:46,284 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:46,283 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:54:46,284 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:54:46,284 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:46,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:46,284 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:46,284 DEBUG [Thread-1624] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:54:46,284 DEBUG [Thread-1624] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:54:46,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:46,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:46,384 DEBUG [Thread-1624] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:54:46,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:54:46,386 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:54:46,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:54:46,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:54:46,388 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:54:46,388 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:46,388 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:54:46,388 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:54:46,389 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:46,389 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:46,390 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:46,390 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:46,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:46,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:46,391 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:46,391 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:54:46,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:46,391 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:46,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:54:46,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:46,391 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:54:46,392 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:54:46,392 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:54:46,392 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:54:46,392 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:46,392 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:54:46,392 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:54:46,392 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:54:46,392 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. 2023-04-19 20:54:46,392 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. started... 2023-04-19 20:54:46,392 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 09bc544118dfd67701ac39d6da1a53f4 3/3 column families, dataSize=140.89 KB heapSize=399.38 KB 2023-04-19 20:54:46,392 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09bc544118dfd67701ac39d6da1a53f4, store=A 2023-04-19 20:54:46,393 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 20:54:46,393 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09bc544118dfd67701ac39d6da1a53f4, store=B 2023-04-19 20:54:46,393 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 20:54:46,393 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09bc544118dfd67701ac39d6da1a53f4, store=C 2023-04-19 20:54:46,393 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 20:54:46,414 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=64 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/A/1420f10daa8a47d28ce9cad6985d6c44 2023-04-19 20:54:46,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(9158): Flush requested on 09bc544118dfd67701ac39d6da1a53f4 2023-04-19 20:54:46,558 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. as already flushing 2023-04-19 20:54:46,565 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:46,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34572 deadline: 1681937746562, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:46,565 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:46,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34548 deadline: 1681937746562, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:46,568 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:46,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34562 deadline: 1681937746565, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:46,599 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:46,599 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:46,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34552 deadline: 1681937746597, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:46,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34540 deadline: 1681937746596, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:46,669 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:46,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34572 deadline: 1681937746667, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:46,669 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:46,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34548 deadline: 1681937746667, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:46,670 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:46,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34562 deadline: 1681937746669, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:46,830 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=64 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/B/1bb1be621f6e434ab49bef17e8960cfa 2023-04-19 20:54:46,872 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:46,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34572 deadline: 1681937746871, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:46,873 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:46,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34548 deadline: 1681937746871, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:46,874 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:46,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34562 deadline: 1681937746872, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:47,177 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:47,177 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:47,178 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-19 20:54:47,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34548 deadline: 1681937747175, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:47,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34572 deadline: 1681937747175, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:47,178 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:47,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34562 deadline: 1681937747176, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:47,246 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=64 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/C/99e8bfc6626a4015a341c31dce1ebbb9 2023-04-19 20:54:47,251 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/A/1420f10daa8a47d28ce9cad6985d6c44 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/1420f10daa8a47d28ce9cad6985d6c44 2023-04-19 20:54:47,255 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/1420f10daa8a47d28ce9cad6985d6c44, entries=150, sequenceid=64, filesize=11.6 K 2023-04-19 20:54:47,256 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/B/1bb1be621f6e434ab49bef17e8960cfa as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/1bb1be621f6e434ab49bef17e8960cfa 2023-04-19 20:54:47,261 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/1bb1be621f6e434ab49bef17e8960cfa, entries=150, sequenceid=64, filesize=11.6 K 2023-04-19 20:54:47,264 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/C/99e8bfc6626a4015a341c31dce1ebbb9 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/99e8bfc6626a4015a341c31dce1ebbb9 2023-04-19 20:54:47,270 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/99e8bfc6626a4015a341c31dce1ebbb9, entries=150, sequenceid=64, filesize=11.6 K 2023-04-19 20:54:47,271 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~399.33 KB/408912, currentSize=40.25 KB/41220 for 09bc544118dfd67701ac39d6da1a53f4 in 879ms, sequenceid=64, compaction requested=false 2023-04-19 20:54:47,271 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 09bc544118dfd67701ac39d6da1a53f4: 2023-04-19 20:54:47,271 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. 2023-04-19 20:54:47,271 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:54:47,271 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:54:47,271 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:47,271 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:54:47,271 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:54:47,273 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:47,273 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:47,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:47,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:47,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:47,274 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:47,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:47,274 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:54:47,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:47,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:47,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:47,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:47,275 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:54:47,275 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@770552e6[Count = 0] remaining members to acquire global barrier 2023-04-19 20:54:47,275 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:54:47,275 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:47,277 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:47,277 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:47,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:47,277 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:54:47,277 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:54:47,277 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:54:47,277 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:47,277 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:54:47,284 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:54:47,285 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:47,284 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:47,285 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:54:47,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:47,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:47,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:47,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:47,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:47,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:47,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:47,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:47,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:47,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:47,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:54:47,287 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:54:47,287 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:54:47,287 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:54:47,287 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:54:47,287 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:54:47,287 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:54:47,289 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:47,289 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:47,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:47,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:47,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:47,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:47,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:47,289 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:47,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:47,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:47,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:47,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:47,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:47,290 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:47,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:47,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:47,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:47,292 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:47,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:47,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:47,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:47,294 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:47,294 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:47,294 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:47,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:47,294 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:54:47,294 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:54:47,294 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:47,294 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:47,294 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:47,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:47,294 DEBUG [Thread-1624] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:54:47,294 DEBUG [Thread-1624] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:54:47,294 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:47,295 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:47,295 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:47,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:47,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:47,295 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:47,395 DEBUG [Thread-1624] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:54:47,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:54:47,397 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:54:47,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:54:47,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:54:47,401 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:54:47,401 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:47,401 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:54:47,401 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:54:47,401 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:47,401 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:47,402 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:47,402 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:47,402 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:47,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:47,403 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:47,403 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:54:47,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:47,403 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:47,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:54:47,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:47,403 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:54:47,404 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:54:47,404 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:54:47,404 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:54:47,404 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:47,404 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:54:47,404 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:54:47,404 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:54:47,404 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. 2023-04-19 20:54:47,404 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. started... 2023-04-19 20:54:47,404 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 09bc544118dfd67701ac39d6da1a53f4 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-19 20:54:47,404 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09bc544118dfd67701ac39d6da1a53f4, store=A 2023-04-19 20:54:47,404 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:54:47,404 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09bc544118dfd67701ac39d6da1a53f4, store=B 2023-04-19 20:54:47,404 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:54:47,404 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09bc544118dfd67701ac39d6da1a53f4, store=C 2023-04-19 20:54:47,405 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:54:47,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(9158): Flush requested on 09bc544118dfd67701ac39d6da1a53f4 2023-04-19 20:54:47,603 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. as already flushing 2023-04-19 20:54:47,639 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681937687627, max timestamp=1681937687637Num uniques 859; 2023-04-19 20:54:47,639 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681937687627, max timestamp=1681937687637Num uniques 859; 2023-04-19 20:54:47,639 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 20:54:47,639 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681937687627, max timestamp=1681937687637Num uniques 859; 2023-04-19 20:54:47,639 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 20:54:47,639 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 20:54:47,675 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681937687662, max timestamp=1681937687673Num uniques 1018; 2023-04-19 20:54:47,675 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-19 20:54:47,676 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7684, totalHeapSize=17712, min timestamp=1681937687662, max timestamp=1681937687673Num uniques 1018; 2023-04-19 20:54:47,676 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46614, new segment data size=7684, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-19 20:54:47,676 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7682, totalHeapSize=17712, min timestamp=1681937687662, max timestamp=1681937687673Num uniques 1018; 2023-04-19 20:54:47,676 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46612, new segment data size=7682, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-19 20:54:47,693 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:47,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34548 deadline: 1681937747690, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:47,695 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1681937687689, max timestamp=1681937687692Num uniques 1027; 2023-04-19 20:54:47,695 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1681937687689, max timestamp=1681937687692Num uniques 1027; 2023-04-19 20:54:47,695 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10392, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-04-19 20:54:47,695 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1681937687689, max timestamp=1681937687692Num uniques 1027; 2023-04-19 20:54:47,695 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10392, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-04-19 20:54:47,695 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10392, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-04-19 20:54:47,734 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:47,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34540 deadline: 1681937747710, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:47,736 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=286, cellSize=13096, totalHeapSize=29984, min timestamp=1681937687707, max timestamp=1681937687717Num uniques 1086; 2023-04-19 20:54:47,736 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=13096, suffix heap size=140912,new segment heap size=29984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=286 2023-04-19 20:54:47,736 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681937687708, max timestamp=1681937687717Num uniques 1086; 2023-04-19 20:54:47,736 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=6870, suffix heap size=140912,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=150 2023-04-19 20:54:47,736 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=286, cellSize=13096, totalHeapSize=29984, min timestamp=1681937687707, max timestamp=1681937687717Num uniques 1086; 2023-04-19 20:54:47,736 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=13096, suffix heap size=140912,new segment heap size=29984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=286 2023-04-19 20:54:47,768 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:47,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34572 deadline: 1681937747765, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:47,768 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:47,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34562 deadline: 1681937747766, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:47,768 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:47,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34552 deadline: 1681937747766, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:47,795 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:47,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34548 deadline: 1681937747794, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:47,813 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=73 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/A/c57b1ce9282947dc8326ed3b2ea0670e 2023-04-19 20:54:47,837 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:47,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34540 deadline: 1681937747836, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:47,871 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:47,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34572 deadline: 1681937747869, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:47,872 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:47,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34562 deadline: 1681937747869, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:47,872 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:47,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34552 deadline: 1681937747869, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:47,998 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:47,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34548 deadline: 1681937747997, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:48,040 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:48,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34540 deadline: 1681937748039, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:48,075 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:48,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34572 deadline: 1681937748073, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:48,075 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:48,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34562 deadline: 1681937748074, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:48,076 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:48,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34552 deadline: 1681937748074, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:48,225 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=73 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/B/6dcfc11e83034a8dae373c2c552ab327 2023-04-19 20:54:48,301 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:48,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34548 deadline: 1681937748301, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:48,344 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:48,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34540 deadline: 1681937748341, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:48,377 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:48,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34572 deadline: 1681937748376, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:48,377 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:48,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34562 deadline: 1681937748376, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:48,379 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:48,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34552 deadline: 1681937748377, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:48,636 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=73 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/C/94abbcc9eebe4d6f9e908f6c71835d25 2023-04-19 20:54:48,640 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/A/c57b1ce9282947dc8326ed3b2ea0670e as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/c57b1ce9282947dc8326ed3b2ea0670e 2023-04-19 20:54:48,644 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/c57b1ce9282947dc8326ed3b2ea0670e, entries=150, sequenceid=73, filesize=11.6 K 2023-04-19 20:54:48,645 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/B/6dcfc11e83034a8dae373c2c552ab327 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/6dcfc11e83034a8dae373c2c552ab327 2023-04-19 20:54:48,648 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/6dcfc11e83034a8dae373c2c552ab327, entries=150, sequenceid=73, filesize=11.6 K 2023-04-19 20:54:48,649 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/C/94abbcc9eebe4d6f9e908f6c71835d25 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/94abbcc9eebe4d6f9e908f6c71835d25 2023-04-19 20:54:48,653 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/94abbcc9eebe4d6f9e908f6c71835d25, entries=150, sequenceid=73, filesize=11.6 K 2023-04-19 20:54:48,653 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=141.52 KB/144914 for 09bc544118dfd67701ac39d6da1a53f4 in 1249ms, sequenceid=73, compaction requested=true 2023-04-19 20:54:48,653 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 09bc544118dfd67701ac39d6da1a53f4: 2023-04-19 20:54:48,654 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. 2023-04-19 20:54:48,654 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:54:48,654 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:54:48,654 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:48,654 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:54:48,654 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:54:48,656 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:48,656 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:48,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:48,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:48,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:48,656 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:48,656 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:54:48,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:48,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:48,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:48,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:48,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:48,657 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:54:48,658 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1ebc21cb[Count = 0] remaining members to acquire global barrier 2023-04-19 20:54:48,658 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:54:48,658 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:48,659 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:48,659 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:48,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:48,659 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:54:48,659 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:54:48,659 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:54:48,660 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:48,660 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:54:48,661 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:54:48,661 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:48,661 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:48,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:48,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:48,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:48,661 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:54:48,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:48,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:48,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:48,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:48,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:48,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:48,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:48,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:54:48,663 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:54:48,663 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:54:48,663 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:54:48,663 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:54:48,663 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:54:48,663 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:54:48,666 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:48,666 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:48,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:48,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:48,666 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:48,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:48,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:48,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:48,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:48,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:48,666 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:48,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:48,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:48,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:48,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:48,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:48,668 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:48,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:48,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:48,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:48,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:48,670 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:48,670 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:48,670 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:48,670 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:48,670 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:48,670 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:48,670 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:48,670 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:48,670 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:48,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:48,670 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:48,671 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:48,671 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:54:48,671 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:54:48,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:48,672 DEBUG [Thread-1624] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:54:48,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:48,671 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:54:48,672 DEBUG [Thread-1624] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:54:48,772 DEBUG [Thread-1624] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:54:48,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:54:48,774 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:54:48,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:54:48,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:54:48,775 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:54:48,775 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:48,775 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:54:48,775 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:54:48,776 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:48,776 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:48,779 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:48,779 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:48,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:48,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:48,779 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:48,779 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:54:48,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:48,779 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:48,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:54:48,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:48,780 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:54:48,780 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:54:48,780 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:54:48,780 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:54:48,780 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:48,780 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:54:48,780 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:54:48,780 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:54:48,780 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. 2023-04-19 20:54:48,780 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. started... 2023-04-19 20:54:48,781 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 09bc544118dfd67701ac39d6da1a53f4 3/3 column families, dataSize=141.52 KB heapSize=399.11 KB 2023-04-19 20:54:48,781 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09bc544118dfd67701ac39d6da1a53f4, store=A 2023-04-19 20:54:48,781 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 20:54:48,781 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09bc544118dfd67701ac39d6da1a53f4, store=B 2023-04-19 20:54:48,781 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 20:54:48,781 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09bc544118dfd67701ac39d6da1a53f4, store=C 2023-04-19 20:54:48,781 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 20:54:48,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(9158): Flush requested on 09bc544118dfd67701ac39d6da1a53f4 2023-04-19 20:54:48,808 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. as already flushing 2023-04-19 20:54:48,830 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:48,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34548 deadline: 1681937748827, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:48,850 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:48,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34540 deadline: 1681937748849, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:48,879 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:48,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34562 deadline: 1681937748878, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:48,882 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:48,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34552 deadline: 1681937748881, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:48,884 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:48,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34572 deadline: 1681937748882, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:48,932 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:48,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34548 deadline: 1681937748931, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:49,136 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:49,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34548 deadline: 1681937749134, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:49,189 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=161 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/A/2958104446344d619c27f6255da723e5 2023-04-19 20:54:49,438 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:49,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34548 deadline: 1681937749438, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:49,600 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=161 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/B/0492788529c84900b44105b6f81a53ad 2023-04-19 20:54:49,859 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:49,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34540 deadline: 1681937749858, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:49,883 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:49,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34562 deadline: 1681937749882, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:49,888 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:49,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34572 deadline: 1681937749887, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:49,891 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:49,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34552 deadline: 1681937749889, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:49,941 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:49,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34548 deadline: 1681937749939, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:50,011 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=43.12 KB at sequenceid=161 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/C/17013f7f119840da8a2a16ea307d7997 2023-04-19 20:54:50,016 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/A/2958104446344d619c27f6255da723e5 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/2958104446344d619c27f6255da723e5 2023-04-19 20:54:50,020 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/2958104446344d619c27f6255da723e5, entries=150, sequenceid=161, filesize=11.8 K 2023-04-19 20:54:50,020 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/B/0492788529c84900b44105b6f81a53ad as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/0492788529c84900b44105b6f81a53ad 2023-04-19 20:54:50,024 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/0492788529c84900b44105b6f81a53ad, entries=150, sequenceid=161, filesize=11.8 K 2023-04-19 20:54:50,025 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/C/17013f7f119840da8a2a16ea307d7997 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/17013f7f119840da8a2a16ea307d7997 2023-04-19 20:54:50,029 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/17013f7f119840da8a2a16ea307d7997, entries=150, sequenceid=161, filesize=11.8 K 2023-04-19 20:54:50,029 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~141.52 KB/144914, heapSize ~399.06 KB/408640, currentSize=40.25 KB/41220 for 09bc544118dfd67701ac39d6da1a53f4 in 1248ms, sequenceid=161, compaction requested=true 2023-04-19 20:54:50,029 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 09bc544118dfd67701ac39d6da1a53f4: 2023-04-19 20:54:50,029 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. 2023-04-19 20:54:50,030 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:54:50,030 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:54:50,030 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:50,030 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:54:50,030 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:54:50,032 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:50,032 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:50,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:50,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:50,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:50,033 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:50,033 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:54:50,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:50,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:50,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:50,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:50,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:50,034 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:54:50,034 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@466fbb5d[Count = 0] remaining members to acquire global barrier 2023-04-19 20:54:50,034 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:54:50,034 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:50,035 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:50,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:50,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:50,036 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:54:50,036 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:54:50,036 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:54:50,036 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:50,036 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:54:50,037 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:50,037 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:54:50,037 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:50,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:50,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:50,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:50,037 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:54:50,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:50,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:50,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:50,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:50,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:50,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:50,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:50,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:54:50,039 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:54:50,039 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:54:50,039 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:54:50,039 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:54:50,039 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:54:50,039 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:54:50,041 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:50,041 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:50,041 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:50,041 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:50,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:50,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:50,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:50,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:50,041 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:50,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:50,042 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:50,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:50,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:50,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:50,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:50,043 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:50,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:50,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:50,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:50,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:50,045 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:50,045 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:50,046 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:50,046 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:50,046 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:50,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:50,046 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:54:50,046 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:54:50,046 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:50,046 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:50,046 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:50,046 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:50,046 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:50,046 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:50,046 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:50,046 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:54:50,046 DEBUG [Thread-1624] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:54:50,046 DEBUG [Thread-1624] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:54:50,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:50,146 DEBUG [Thread-1624] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:54:50,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:54:50,149 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:54:50,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:54:50,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:54:50,150 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:54:50,150 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:50,150 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:54:50,150 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:54:50,151 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:50,151 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:50,152 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:50,152 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:50,152 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:50,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:50,152 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:50,152 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:54:50,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:50,153 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:50,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:54:50,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:50,153 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:54:50,153 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:54:50,153 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:54:50,153 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:54:50,153 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:50,153 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:54:50,153 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:54:50,153 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:54:50,153 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. 2023-04-19 20:54:50,153 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. started... 2023-04-19 20:54:50,154 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 09bc544118dfd67701ac39d6da1a53f4 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-19 20:54:50,154 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09bc544118dfd67701ac39d6da1a53f4, store=A 2023-04-19 20:54:50,154 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:54:50,154 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09bc544118dfd67701ac39d6da1a53f4, store=B 2023-04-19 20:54:50,154 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:54:50,154 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09bc544118dfd67701ac39d6da1a53f4, store=C 2023-04-19 20:54:50,154 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:54:50,561 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=170 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/A/df24c00b109c45d99964454a1fbbdca2 2023-04-19 20:54:50,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(9158): Flush requested on 09bc544118dfd67701ac39d6da1a53f4 2023-04-19 20:54:50,954 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. as already flushing 2023-04-19 20:54:50,977 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=170 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/B/d1c36bdb21ac4a6b8f682ba8ded71415 2023-04-19 20:54:51,013 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681937690995, max timestamp=1681937691011Num uniques 859; 2023-04-19 20:54:51,013 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 20:54:51,013 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681937690995, max timestamp=1681937691011Num uniques 859; 2023-04-19 20:54:51,014 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 20:54:51,014 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681937690995, max timestamp=1681937691011Num uniques 859; 2023-04-19 20:54:51,014 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 20:54:51,085 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681937691066, max timestamp=1681937691082Num uniques 1018; 2023-04-19 20:54:51,085 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-19 20:54:51,085 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681937691066, max timestamp=1681937691082Num uniques 1018; 2023-04-19 20:54:51,085 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681937691066, max timestamp=1681937691082Num uniques 1018; 2023-04-19 20:54:51,085 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-19 20:54:51,085 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-19 20:54:51,142 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1681937691125, max timestamp=1681937691139Num uniques 1027; 2023-04-19 20:54:51,142 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-19 20:54:51,142 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1681937691125, max timestamp=1681937691139Num uniques 1027; 2023-04-19 20:54:51,142 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1681937691125, max timestamp=1681937691139Num uniques 1027; 2023-04-19 20:54:51,142 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-19 20:54:51,142 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-19 20:54:51,200 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1681937691176, max timestamp=1681937691198Num uniques 1036; 2023-04-19 20:54:51,200 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1681937691176, max timestamp=1681937691198Num uniques 1036; 2023-04-19 20:54:51,201 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-04-19 20:54:51,201 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1681937691176, max timestamp=1681937691198Num uniques 1036; 2023-04-19 20:54:51,201 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-04-19 20:54:51,201 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-04-19 20:54:51,262 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1681937691228, max timestamp=1681937691260Num uniques 1045; 2023-04-19 20:54:51,262 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-04-19 20:54:51,262 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1681937691228, max timestamp=1681937691260Num uniques 1045; 2023-04-19 20:54:51,262 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-04-19 20:54:51,263 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1681937691228, max timestamp=1681937691260Num uniques 1045; 2023-04-19 20:54:51,263 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-04-19 20:54:51,325 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:51,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34548 deadline: 1681937751322, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:51,387 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=170 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/C/56b9a028ace1448ca126392a183aa92e 2023-04-19 20:54:51,392 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/A/df24c00b109c45d99964454a1fbbdca2 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/df24c00b109c45d99964454a1fbbdca2 2023-04-19 20:54:51,396 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/df24c00b109c45d99964454a1fbbdca2, entries=150, sequenceid=170, filesize=11.8 K 2023-04-19 20:54:51,397 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/B/d1c36bdb21ac4a6b8f682ba8ded71415 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/d1c36bdb21ac4a6b8f682ba8ded71415 2023-04-19 20:54:51,400 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/d1c36bdb21ac4a6b8f682ba8ded71415, entries=150, sequenceid=170, filesize=11.8 K 2023-04-19 20:54:51,403 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/C/56b9a028ace1448ca126392a183aa92e as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/56b9a028ace1448ca126392a183aa92e 2023-04-19 20:54:51,407 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/56b9a028ace1448ca126392a183aa92e, entries=150, sequenceid=170, filesize=11.8 K 2023-04-19 20:54:51,408 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=140.89 KB/144270 for 09bc544118dfd67701ac39d6da1a53f4 in 1254ms, sequenceid=170, compaction requested=true 2023-04-19 20:54:51,408 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 09bc544118dfd67701ac39d6da1a53f4: 2023-04-19 20:54:51,409 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. 2023-04-19 20:54:51,409 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:54:51,409 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:54:51,409 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:51,413 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:54:51,413 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:54:51,415 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:51,415 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:51,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:51,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:51,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:51,416 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:51,416 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:54:51,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:51,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:51,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:51,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:51,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:51,418 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:54:51,418 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7b7afbe1[Count = 0] remaining members to acquire global barrier 2023-04-19 20:54:51,418 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:54:51,418 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:51,419 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:51,419 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:51,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:51,419 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:54:51,419 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:54:51,419 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:54:51,419 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:51,419 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:54:51,421 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:54:51,421 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:51,421 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:54:51,421 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:51,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:51,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:51,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:51,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:51,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:51,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:51,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:51,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:51,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:51,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:51,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:54:51,424 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:54:51,424 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:54:51,424 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:54:51,424 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:54:51,424 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:54:51,424 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:54:51,426 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:51,426 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:51,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:51,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:51,426 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:51,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:51,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:51,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:51,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:51,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:51,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:51,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:51,427 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:51,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:51,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:51,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:51,429 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:51,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:51,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:51,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:51,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:51,431 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:51,432 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:51,432 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:51,432 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:51,432 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:54:51,432 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:54:51,432 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:51,432 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:51,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:51,432 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:51,432 DEBUG [Thread-1624] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:54:51,432 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:51,432 DEBUG [Thread-1624] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:54:51,432 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:54:51,432 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:51,432 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:51,432 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:51,432 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:51,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:51,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(9158): Flush requested on 09bc544118dfd67701ac39d6da1a53f4 2023-04-19 20:54:51,433 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 09bc544118dfd67701ac39d6da1a53f4 3/3 column families, dataSize=147.60 KB heapSize=422.16 KB 2023-04-19 20:54:51,433 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09bc544118dfd67701ac39d6da1a53f4, store=A 2023-04-19 20:54:51,433 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-19 20:54:51,433 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09bc544118dfd67701ac39d6da1a53f4, store=B 2023-04-19 20:54:51,434 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-19 20:54:51,434 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09bc544118dfd67701ac39d6da1a53f4, store=C 2023-04-19 20:54:51,434 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-19 20:54:51,454 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=277 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/A/dd84fac044094914b5f1ff40bea933de 2023-04-19 20:54:51,468 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:51,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34548 deadline: 1681937751465, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:51,532 DEBUG [Thread-1624] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:54:51,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:54:51,534 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:54:51,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:54:51,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:54:51,536 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:54:51,536 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:51,536 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:54:51,536 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:54:51,536 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:51,536 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:51,538 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:51,538 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:51,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:51,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:51,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:51,538 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:51,538 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:54:51,538 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:51,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:54:51,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:51,539 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:54:51,539 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:54:51,539 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:54:51,539 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:54:51,539 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:51,539 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:54:51,539 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:54:51,539 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:54:51,539 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. 2023-04-19 20:54:51,539 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. started... 2023-04-19 20:54:51,540 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. as already flushing 2023-04-19 20:54:51,540 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. 2023-04-19 20:54:51,540 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:54:51,540 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:54:51,540 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:51,540 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:54:51,540 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:54:51,545 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:51,545 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:51,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:51,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:51,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:51,545 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:51,545 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:54:51,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:51,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:51,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:51,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:51,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:51,546 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:54:51,547 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2635140a[Count = 0] remaining members to acquire global barrier 2023-04-19 20:54:51,547 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:54:51,547 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:51,548 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:51,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:51,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:51,548 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:54:51,548 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:54:51,548 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:54:51,548 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:51,548 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:54:51,549 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:51,549 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:54:51,549 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:51,549 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:54:51,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:51,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:51,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:51,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:51,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:51,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:51,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:51,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:51,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:51,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:51,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:54:51,552 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:54:51,552 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:54:51,552 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:54:51,552 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:54:51,552 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:54:51,552 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:54:51,554 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:51,554 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:51,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:51,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:51,554 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:51,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:51,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:51,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:51,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:51,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:51,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:51,554 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:51,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:51,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:51,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:51,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:51,556 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:51,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:51,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:51,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:51,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:51,559 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:51,559 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:51,559 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:51,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:51,559 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:54:51,559 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:54:51,559 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:51,559 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:51,559 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:54:51,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:51,559 DEBUG [Thread-1624] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:54:51,559 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:51,559 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:51,559 DEBUG [Thread-1624] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:54:51,559 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:51,559 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:51,560 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:51,560 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:51,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:51,570 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:51,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34548 deadline: 1681937751569, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:51,660 DEBUG [Thread-1624] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:54:51,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:54:51,662 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:54:51,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:54:51,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:54:51,664 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:54:51,664 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:51,664 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:54:51,664 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:54:51,665 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:51,665 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:51,666 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:51,666 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:51,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:51,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:51,666 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:51,666 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:54:51,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:51,667 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:51,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:54:51,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:51,667 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:54:51,667 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:54:51,667 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:54:51,667 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:54:51,667 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:51,667 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:54:51,667 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:54:51,668 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:54:51,668 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. 2023-04-19 20:54:51,668 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. started... 2023-04-19 20:54:51,668 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. as already flushing 2023-04-19 20:54:51,668 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. 2023-04-19 20:54:51,668 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:54:51,668 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:54:51,668 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:51,668 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:54:51,668 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:54:51,670 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:51,670 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:51,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:51,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:51,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:51,670 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:51,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:51,670 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:54:51,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:51,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:51,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:51,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:51,671 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:54:51,671 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5978db7e[Count = 0] remaining members to acquire global barrier 2023-04-19 20:54:51,671 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:54:51,671 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:51,675 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:51,676 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:51,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:51,676 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:54:51,676 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:51,676 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:54:51,676 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:54:51,676 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:54:51,677 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:51,677 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:54:51,677 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:51,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:51,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:51,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:51,677 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:54:51,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:51,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:51,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:51,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:51,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:51,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:51,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:51,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:54:51,680 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:54:51,680 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:54:51,680 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:54:51,680 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:54:51,680 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:54:51,680 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:54:51,682 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:51,682 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:51,682 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:51,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:51,682 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:51,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:51,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:51,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:51,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:51,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:51,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:51,683 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:51,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:51,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:51,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:51,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:51,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:51,684 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:51,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:51,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:51,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:51,687 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:51,687 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:51,687 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:51,687 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:51,687 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:51,687 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:54:51,687 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:54:51,687 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:51,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:51,687 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:51,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:51,687 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:51,687 DEBUG [Thread-1624] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:54:51,687 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:51,687 DEBUG [Thread-1624] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:54:51,687 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:51,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:51,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:51,774 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:51,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34548 deadline: 1681937751771, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:51,787 DEBUG [Thread-1624] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:54:51,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:54:51,789 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:54:51,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:54:51,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:54:51,790 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:54:51,791 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:51,791 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:54:51,791 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:54:51,791 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:51,791 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:51,794 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:51,794 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:51,794 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:51,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:51,794 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:51,794 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:54:51,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:51,795 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:51,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:54:51,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:51,795 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:54:51,795 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:54:51,795 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:54:51,795 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:54:51,795 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:51,796 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:54:51,796 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:54:51,796 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:54:51,796 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. 2023-04-19 20:54:51,796 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. started... 2023-04-19 20:54:51,796 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. as already flushing 2023-04-19 20:54:51,796 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. 2023-04-19 20:54:51,796 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:54:51,796 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:54:51,796 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:51,796 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:54:51,796 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:54:51,799 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:51,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:51,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:51,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:51,799 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:51,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:51,799 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:51,799 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:54:51,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:51,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:51,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:51,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:51,801 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:54:51,801 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6276265f[Count = 0] remaining members to acquire global barrier 2023-04-19 20:54:51,801 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:54:51,801 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:51,803 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:51,803 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:51,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:51,803 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:54:51,803 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:54:51,803 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:54:51,803 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:51,803 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:54:51,805 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:54:51,805 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:51,805 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:51,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:51,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:51,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:51,805 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:54:51,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:51,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:51,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:51,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:51,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:51,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:51,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:51,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:54:51,808 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:54:51,808 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:54:51,808 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:54:51,808 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:54:51,808 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:54:51,808 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:54:51,810 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:51,810 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:51,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:51,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:51,811 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:51,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:51,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:51,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:51,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:51,811 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:51,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:51,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:51,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:51,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:51,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:51,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:51,812 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:51,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:51,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:51,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:51,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:51,815 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:51,815 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:51,815 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:51,815 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:51,815 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:54:51,815 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:51,815 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:54:51,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:51,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:51,815 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:51,815 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:51,815 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:51,815 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:51,816 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:51,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:51,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:51,816 DEBUG [Thread-1624] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:54:51,816 DEBUG [Thread-1624] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:54:51,866 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=277 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/B/658221b251bc41e08cdddc0e79797404 2023-04-19 20:54:51,873 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:51,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34540 deadline: 1681937751871, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:51,875 DEBUG [Thread-1618] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4164 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4., hostname=jenkins-hbase4.apache.org,35247,1681937616971, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-19 20:54:51,895 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:51,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34552 deadline: 1681937751892, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:51,896 DEBUG [Thread-1616] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4129 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4., hostname=jenkins-hbase4.apache.org,35247,1681937616971, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-19 20:54:51,897 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:51,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34562 deadline: 1681937751894, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:51,898 DEBUG [Thread-1622] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4131 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4., hostname=jenkins-hbase4.apache.org,35247,1681937616971, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-19 20:54:51,899 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:51,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34572 deadline: 1681937751896, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:51,900 DEBUG [Thread-1614] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4134 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4., hostname=jenkins-hbase4.apache.org,35247,1681937616971, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-19 20:54:51,916 DEBUG [Thread-1624] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:54:51,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:54:51,918 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:54:51,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:54:51,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:54:51,920 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:54:51,920 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:51,920 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:54:51,920 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:54:51,921 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:51,921 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:51,922 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:51,922 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:51,922 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:51,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:51,922 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:51,922 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:54:51,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:51,923 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:51,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:54:51,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:51,923 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:54:51,923 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:54:51,924 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:54:51,924 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:54:51,924 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:51,924 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:54:51,924 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:54:51,924 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:54:51,924 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. 2023-04-19 20:54:51,924 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. started... 2023-04-19 20:54:51,924 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. as already flushing 2023-04-19 20:54:51,924 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. 2023-04-19 20:54:51,924 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:54:51,924 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:54:51,924 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:51,924 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:54:51,924 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:54:51,926 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:51,926 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:51,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:51,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:51,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:51,926 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:51,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:51,926 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:54:51,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:51,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:51,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:51,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:51,927 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:54:51,927 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4eff7c03[Count = 0] remaining members to acquire global barrier 2023-04-19 20:54:51,927 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:54:51,928 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:51,929 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:51,930 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:51,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:51,930 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:54:51,930 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:54:51,930 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:54:51,930 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:51,930 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:54:51,931 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:51,931 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:54:51,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:51,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:51,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:51,931 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:51,931 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:54:51,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:51,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:51,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:51,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:51,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:51,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:51,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:51,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:54:51,933 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:54:51,934 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:54:51,934 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:54:51,934 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:54:51,934 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:54:51,934 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:54:51,935 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:51,935 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:51,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:51,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:51,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:51,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:51,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:51,935 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:51,936 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:51,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:51,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:51,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:51,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:51,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:51,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:51,937 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:51,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:51,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:51,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:51,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:51,939 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:51,939 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:51,939 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:51,939 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:51,939 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:51,939 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:51,939 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:54:51,939 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:54:51,939 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:51,939 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:51,939 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:51,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:51,939 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:51,939 DEBUG [Thread-1624] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:54:51,939 DEBUG [Thread-1624] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:54:51,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:51,939 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:54:51,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:51,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:52,039 DEBUG [Thread-1624] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:54:52,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:54:52,042 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:54:52,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:54:52,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:54:52,043 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:54:52,043 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:52,043 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:54:52,043 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:54:52,044 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:52,044 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:52,046 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:52,046 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:52,046 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:52,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:52,046 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:52,047 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:54:52,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:52,047 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:52,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:54:52,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:52,047 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:54:52,048 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:54:52,048 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:54:52,048 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:54:52,048 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:52,048 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:54:52,048 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:54:52,048 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:54:52,048 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. 2023-04-19 20:54:52,048 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. started... 2023-04-19 20:54:52,048 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. as already flushing 2023-04-19 20:54:52,048 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. 2023-04-19 20:54:52,048 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:54:52,048 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:54:52,048 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:52,048 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:54:52,048 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:54:52,050 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:52,050 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:52,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:52,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:52,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:52,050 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:52,050 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:54:52,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:52,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:52,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:52,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:52,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:52,051 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:54:52,051 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@40e4a492[Count = 0] remaining members to acquire global barrier 2023-04-19 20:54:52,051 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:54:52,052 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:52,053 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:52,053 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:52,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:52,053 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:54:52,053 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:54:52,053 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:54:52,053 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:52,053 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:54:52,063 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:52,063 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:54:52,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:52,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:52,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:52,063 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:52,063 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:54:52,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:52,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:52,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:52,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:52,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:52,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:52,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:52,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:54:52,065 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:54:52,065 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:54:52,065 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:54:52,065 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:54:52,065 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:54:52,065 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:54:52,067 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:52,067 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:52,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:52,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:52,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:52,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:52,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:52,067 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:52,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:52,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:52,068 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:52,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:52,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:52,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:52,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:52,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:52,069 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:52,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:52,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:52,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:52,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:52,071 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:52,071 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:52,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:52,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:52,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:52,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:52,072 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:52,072 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:52,072 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:52,072 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:52,072 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:52,072 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:54:52,072 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:54:52,072 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:52,072 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:52,072 DEBUG [Thread-1624] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:54:52,072 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:54:52,072 DEBUG [Thread-1624] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:54:52,072 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:52,077 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:52,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34548 deadline: 1681937752076, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:52,173 DEBUG [Thread-1624] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:54:52,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:54:52,175 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:54:52,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:54:52,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:54:52,176 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:54:52,176 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:52,176 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:54:52,176 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:54:52,176 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:52,177 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:52,178 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:52,178 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:52,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:52,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:52,178 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:52,178 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:54:52,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:52,179 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:52,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:54:52,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:52,179 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:54:52,179 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:54:52,179 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:54:52,179 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:54:52,179 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:52,179 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:54:52,180 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:54:52,180 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:54:52,180 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. 2023-04-19 20:54:52,180 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. started... 2023-04-19 20:54:52,180 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. as already flushing 2023-04-19 20:54:52,180 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. 2023-04-19 20:54:52,180 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:54:52,180 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:54:52,180 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:52,180 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:54:52,180 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:54:52,182 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:52,182 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:52,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:52,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:52,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:52,183 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:52,183 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:54:52,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:52,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:52,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:52,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:52,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:52,184 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:54:52,184 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5f5834db[Count = 0] remaining members to acquire global barrier 2023-04-19 20:54:52,184 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:54:52,184 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:52,185 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:52,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:52,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:52,185 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:54:52,185 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:54:52,185 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:54:52,185 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:52,185 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:54:52,187 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:54:52,187 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:52,187 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:52,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:52,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:52,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:52,187 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:54:52,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:52,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:52,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:52,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:52,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:52,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:52,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:52,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:54:52,189 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:54:52,189 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:54:52,189 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:54:52,189 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:54:52,189 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:54:52,189 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:54:52,191 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:52,191 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:52,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:52,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:52,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:52,191 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:52,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:52,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:52,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:52,191 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:52,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:52,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:52,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:52,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:52,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:52,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:52,193 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:52,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:52,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:52,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:52,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:52,196 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:52,196 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:52,196 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:52,196 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:52,196 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:52,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:52,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:52,196 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:52,196 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:54:52,197 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:54:52,196 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:52,197 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:52,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:52,197 DEBUG [Thread-1624] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:54:52,197 DEBUG [Thread-1624] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:54:52,197 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:52,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:52,197 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:52,282 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=277 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/C/7a7a84c1f3a148d18a3938f90ea1a0fd 2023-04-19 20:54:52,286 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/A/dd84fac044094914b5f1ff40bea933de as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/dd84fac044094914b5f1ff40bea933de 2023-04-19 20:54:52,290 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/dd84fac044094914b5f1ff40bea933de, entries=200, sequenceid=277, filesize=14.3 K 2023-04-19 20:54:52,290 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/B/658221b251bc41e08cdddc0e79797404 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/658221b251bc41e08cdddc0e79797404 2023-04-19 20:54:52,294 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/658221b251bc41e08cdddc0e79797404, entries=150, sequenceid=277, filesize=11.9 K 2023-04-19 20:54:52,295 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/C/7a7a84c1f3a148d18a3938f90ea1a0fd as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/7a7a84c1f3a148d18a3938f90ea1a0fd 2023-04-19 20:54:52,297 DEBUG [Thread-1624] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:54:52,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:54:52,300 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/7a7a84c1f3a148d18a3938f90ea1a0fd, entries=150, sequenceid=277, filesize=11.9 K 2023-04-19 20:54:52,300 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:54:52,300 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~422.11 KB/432240, currentSize=33.54 KB/34350 for 09bc544118dfd67701ac39d6da1a53f4 in 867ms, sequenceid=277, compaction requested=true 2023-04-19 20:54:52,301 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 09bc544118dfd67701ac39d6da1a53f4: 2023-04-19 20:54:52,301 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 20:54:52,301 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 20:54:52,301 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-19 20:54:52,301 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 20:54:52,301 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-19 20:54:52,303 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 74356 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-19 20:54:52,303 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 71916 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-19 20:54:52,303 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1912): 09bc544118dfd67701ac39d6da1a53f4/A is initiating minor compaction (all files) 2023-04-19 20:54:52,303 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1912): 09bc544118dfd67701ac39d6da1a53f4/B is initiating minor compaction (all files) 2023-04-19 20:54:52,303 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 09bc544118dfd67701ac39d6da1a53f4/A in TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. 2023-04-19 20:54:52,303 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 09bc544118dfd67701ac39d6da1a53f4/B in TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. 2023-04-19 20:54:52,303 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/f6063291209d427bad7d0a0922ab5ac4, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/1420f10daa8a47d28ce9cad6985d6c44, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/c57b1ce9282947dc8326ed3b2ea0670e, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/2958104446344d619c27f6255da723e5, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/df24c00b109c45d99964454a1fbbdca2, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/dd84fac044094914b5f1ff40bea933de] into tmpdir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp, totalSize=72.6 K 2023-04-19 20:54:52,303 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/82f6f8fd358241138162f76c87217e99, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/1bb1be621f6e434ab49bef17e8960cfa, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/6dcfc11e83034a8dae373c2c552ab327, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/0492788529c84900b44105b6f81a53ad, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/d1c36bdb21ac4a6b8f682ba8ded71415, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/658221b251bc41e08cdddc0e79797404] into tmpdir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp, totalSize=70.2 K 2023-04-19 20:54:52,304 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting f6063291209d427bad7d0a0922ab5ac4, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=10, earliestPutTs=1681937685392 2023-04-19 20:54:52,304 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 82f6f8fd358241138162f76c87217e99, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=10, earliestPutTs=1681937685392 2023-04-19 20:54:52,304 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 1420f10daa8a47d28ce9cad6985d6c44, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=64, earliestPutTs=1681937685471 2023-04-19 20:54:52,304 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 1bb1be621f6e434ab49bef17e8960cfa, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=64, earliestPutTs=1681937685471 2023-04-19 20:54:52,305 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting c57b1ce9282947dc8326ed3b2ea0670e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=73, earliestPutTs=1681937686560 2023-04-19 20:54:52,305 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 6dcfc11e83034a8dae373c2c552ab327, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=73, earliestPutTs=1681937686560 2023-04-19 20:54:52,305 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 2958104446344d619c27f6255da723e5, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=161, earliestPutTs=1681937687760 2023-04-19 20:54:52,305 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 0492788529c84900b44105b6f81a53ad, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=161, earliestPutTs=1681937687760 2023-04-19 20:54:52,305 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting df24c00b109c45d99964454a1fbbdca2, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=170, earliestPutTs=1681937688813 2023-04-19 20:54:52,306 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting d1c36bdb21ac4a6b8f682ba8ded71415, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=170, earliestPutTs=1681937688813 2023-04-19 20:54:52,306 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting dd84fac044094914b5f1ff40bea933de, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=277, earliestPutTs=1681937691300 2023-04-19 20:54:52,306 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 658221b251bc41e08cdddc0e79797404, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=277, earliestPutTs=1681937691300 2023-04-19 20:54:52,319 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] throttle.PressureAwareThroughputController(145): 09bc544118dfd67701ac39d6da1a53f4#B#compaction#216 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 57.69 MB/second 2023-04-19 20:54:52,327 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] throttle.PressureAwareThroughputController(145): 09bc544118dfd67701ac39d6da1a53f4#A#compaction#217 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 57.69 MB/second 2023-04-19 20:54:52,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:54:52,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:54:52,332 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:54:52,332 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:52,332 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:54:52,332 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:54:52,333 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:52,333 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:52,334 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:52,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:52,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:52,334 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:52,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:52,335 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:52,335 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:54:52,335 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:52,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:54:52,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:52,335 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:54:52,335 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:54:52,336 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:54:52,336 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:54:52,336 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:52,336 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:54:52,342 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:54:52,342 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:54:52,342 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. 2023-04-19 20:54:52,342 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. started... 2023-04-19 20:54:52,342 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 09bc544118dfd67701ac39d6da1a53f4 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-19 20:54:52,342 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09bc544118dfd67701ac39d6da1a53f4, store=A 2023-04-19 20:54:52,343 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:54:52,343 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09bc544118dfd67701ac39d6da1a53f4, store=B 2023-04-19 20:54:52,343 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:54:52,343 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09bc544118dfd67701ac39d6da1a53f4, store=C 2023-04-19 20:54:52,343 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:54:52,389 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=285 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/A/bf46b37dae664e82b75afe78db6c2acb 2023-04-19 20:54:52,399 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=285 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/B/bfcf02323c734c7f920ed57c840e7d1e 2023-04-19 20:54:52,410 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=285 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/C/a97c730a972f46ab8a1ee332e28dba03 2023-04-19 20:54:52,413 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/A/bf46b37dae664e82b75afe78db6c2acb as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/bf46b37dae664e82b75afe78db6c2acb 2023-04-19 20:54:52,417 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/bf46b37dae664e82b75afe78db6c2acb, entries=150, sequenceid=285, filesize=11.9 K 2023-04-19 20:54:52,418 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/B/bfcf02323c734c7f920ed57c840e7d1e as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/bfcf02323c734c7f920ed57c840e7d1e 2023-04-19 20:54:52,425 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/bfcf02323c734c7f920ed57c840e7d1e, entries=150, sequenceid=285, filesize=11.9 K 2023-04-19 20:54:52,426 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/C/a97c730a972f46ab8a1ee332e28dba03 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/a97c730a972f46ab8a1ee332e28dba03 2023-04-19 20:54:52,439 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/a97c730a972f46ab8a1ee332e28dba03, entries=150, sequenceid=285, filesize=11.9 K 2023-04-19 20:54:52,442 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=0 B/0 for 09bc544118dfd67701ac39d6da1a53f4 in 100ms, sequenceid=285, compaction requested=true 2023-04-19 20:54:52,442 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 09bc544118dfd67701ac39d6da1a53f4: 2023-04-19 20:54:52,442 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. 2023-04-19 20:54:52,442 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:54:52,442 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:54:52,442 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:52,442 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:54:52,442 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:54:52,445 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:52,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:52,445 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:52,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:52,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:52,446 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:52,446 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:54:52,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:52,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:52,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:52,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:52,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:52,448 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:54:52,448 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@ccfb5b1[Count = 0] remaining members to acquire global barrier 2023-04-19 20:54:52,448 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:54:52,448 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:52,449 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:52,449 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:52,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:52,450 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:54:52,450 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:54:52,450 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:54:52,450 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:52,450 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:54:52,452 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:52,452 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:54:52,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:52,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:52,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:52,452 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:52,452 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:54:52,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:52,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:52,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:52,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:52,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:52,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:52,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:52,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:54:52,455 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:54:52,455 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:54:52,455 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:54:52,455 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:54:52,455 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:54:52,456 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:54:52,457 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:52,457 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:52,457 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:52,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:52,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:52,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:52,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:52,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:52,458 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:52,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:52,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:52,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:52,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:52,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:52,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:52,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:52,459 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:52,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:52,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:52,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:52,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:52,462 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:52,462 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:54:52,462 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:54:52,462 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:52,462 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:54:52,462 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:52,462 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:52,462 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:52,463 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:52,463 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:52,463 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:52,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:52,463 DEBUG [Thread-1624] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:54:52,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:52,463 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:52,463 DEBUG [Thread-1624] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:54:52,463 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:52,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:52,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:52,563 DEBUG [Thread-1624] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:54:52,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:54:52,565 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:54:52,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:54:52,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:54:52,567 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:54:52,567 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:52,567 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:54:52,567 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:54:52,567 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:52,567 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:52,569 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:52,569 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:52,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:52,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:52,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:52,569 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:52,569 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:54:52,570 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:52,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:54:52,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:52,570 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:54:52,570 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:54:52,570 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:54:52,570 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:52,571 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:54:52,571 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:54:52,571 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:54:52,571 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. 2023-04-19 20:54:52,571 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. started... 2023-04-19 20:54:52,571 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 09bc544118dfd67701ac39d6da1a53f4: 2023-04-19 20:54:52,571 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. 2023-04-19 20:54:52,571 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:54:52,571 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:54:52,571 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:52,571 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:54:52,571 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:54:52,573 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:52,573 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:52,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:52,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:52,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:52,574 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:52,574 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:54:52,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:52,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:52,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:52,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:52,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:52,575 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:54:52,576 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3965cf4f[Count = 0] remaining members to acquire global barrier 2023-04-19 20:54:52,576 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:54:52,576 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:52,577 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:52,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:52,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:52,577 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:54:52,577 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:54:52,577 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:54:52,577 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:52,577 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:54:52,579 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:54:52,579 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:52,579 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:52,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:52,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:52,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:52,579 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:54:52,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:52,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:52,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:52,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:52,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:52,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:52,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:52,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:54:52,582 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:54:52,582 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:54:52,582 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:54:52,582 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:54:52,582 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:54:52,582 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:54:52,583 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:52,583 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:52,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:52,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:52,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:52,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:52,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:52,584 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:52,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:52,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:52,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:52,584 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:52,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:52,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:52,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:52,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:52,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:52,586 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:52,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:52,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:52,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:52,594 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:52,594 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:52,594 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:52,594 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:52,594 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:54:52,594 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:52,594 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:54:52,594 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:54:52,594 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:52,594 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:52,594 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:52,594 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:52,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:52,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:52,594 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:52,594 DEBUG [Thread-1624] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:54:52,595 DEBUG [Thread-1624] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:54:52,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:52,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:52,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(9158): Flush requested on 09bc544118dfd67701ac39d6da1a53f4 2023-04-19 20:54:52,606 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 09bc544118dfd67701ac39d6da1a53f4 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-19 20:54:52,607 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09bc544118dfd67701ac39d6da1a53f4, store=A 2023-04-19 20:54:52,607 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:54:52,607 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09bc544118dfd67701ac39d6da1a53f4, store=B 2023-04-19 20:54:52,607 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:54:52,607 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09bc544118dfd67701ac39d6da1a53f4, store=C 2023-04-19 20:54:52,607 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:54:52,617 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=296 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/A/1ba5c75941d74be1b8659e0b8fc6da94 2023-04-19 20:54:52,642 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=296 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/B/3ea8fc98b3cd432d8fa8c22cc0602a52 2023-04-19 20:54:52,665 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=296 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/C/ba02cc56583a4873953ca419c5714aeb 2023-04-19 20:54:52,672 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/A/1ba5c75941d74be1b8659e0b8fc6da94 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/1ba5c75941d74be1b8659e0b8fc6da94 2023-04-19 20:54:52,677 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/1ba5c75941d74be1b8659e0b8fc6da94, entries=200, sequenceid=296, filesize=14.3 K 2023-04-19 20:54:52,678 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/B/3ea8fc98b3cd432d8fa8c22cc0602a52 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/3ea8fc98b3cd432d8fa8c22cc0602a52 2023-04-19 20:54:52,684 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/3ea8fc98b3cd432d8fa8c22cc0602a52, entries=150, sequenceid=296, filesize=11.9 K 2023-04-19 20:54:52,685 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/C/ba02cc56583a4873953ca419c5714aeb as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/ba02cc56583a4873953ca419c5714aeb 2023-04-19 20:54:52,690 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/ba02cc56583a4873953ca419c5714aeb, entries=150, sequenceid=296, filesize=11.9 K 2023-04-19 20:54:52,690 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=67.09 KB/68700 for 09bc544118dfd67701ac39d6da1a53f4 in 84ms, sequenceid=296, compaction requested=true 2023-04-19 20:54:52,691 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 09bc544118dfd67701ac39d6da1a53f4: 2023-04-19 20:54:52,691 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-19 20:54:52,691 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=3), splitQueue=0 2023-04-19 20:54:52,691 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=4), splitQueue=0 2023-04-19 20:54:52,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(9158): Flush requested on 09bc544118dfd67701ac39d6da1a53f4 2023-04-19 20:54:52,693 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 09bc544118dfd67701ac39d6da1a53f4 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-19 20:54:52,693 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09bc544118dfd67701ac39d6da1a53f4, store=A 2023-04-19 20:54:52,694 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:54:52,694 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09bc544118dfd67701ac39d6da1a53f4, store=B 2023-04-19 20:54:52,694 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:54:52,694 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09bc544118dfd67701ac39d6da1a53f4, store=C 2023-04-19 20:54:52,694 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:54:52,697 DEBUG [Thread-1624] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:54:52,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:54:52,701 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:54:52,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:54:52,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:54:52,702 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:54:52,702 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:52,702 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:54:52,702 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:54:52,703 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:52,703 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:52,704 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:52,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:52,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:52,704 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:52,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:52,705 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:52,706 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:52,706 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:54:52,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:54:52,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:52,706 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:54:52,710 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:54:52,710 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:54:52,710 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:54:52,710 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:52,710 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:54:52,711 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:54:52,711 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:54:52,711 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. 2023-04-19 20:54:52,711 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. started... 2023-04-19 20:54:52,711 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. as already flushing 2023-04-19 20:54:52,712 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. 2023-04-19 20:54:52,712 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:54:52,712 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:54:52,712 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:52,712 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:54:52,712 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:54:52,714 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:52,714 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:52,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:52,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:52,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:52,715 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:52,715 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:54:52,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:52,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:52,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:52,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:52,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:52,716 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:54:52,716 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c402cc5[Count = 0] remaining members to acquire global barrier 2023-04-19 20:54:52,716 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:54:52,716 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:52,718 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:52,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:52,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:52,718 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:54:52,718 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:54:52,718 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:54:52,718 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:52,718 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:54:52,721 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:54:52,721 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:52,721 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:54:52,721 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:52,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:52,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:52,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:52,722 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=310 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/A/5461affcf7804820a05692cd1a68292e 2023-04-19 20:54:52,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:52,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:52,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:52,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:52,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:52,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:52,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:52,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:54:52,724 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:54:52,724 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:54:52,724 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:54:52,724 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:54:52,724 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:54:52,724 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:54:52,726 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:52,726 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:52,726 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:52,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:52,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:52,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:52,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:52,726 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:52,727 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:52,727 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:52,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:52,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:52,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:52,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:52,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:52,728 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:52,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:52,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:52,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:52,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:52,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:52,731 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:52,731 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:52,731 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:52,731 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:52,731 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:52,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:52,731 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:54:52,731 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:54:52,731 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:52,731 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:52,731 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:52,731 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:52,731 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:52,731 DEBUG [Thread-1624] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:54:52,731 DEBUG [Thread-1624] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:54:52,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:52,731 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:54:52,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:52,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:52,777 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/B/82eaedbf08294b58830dc03c11c92fd6 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/82eaedbf08294b58830dc03c11c92fd6 2023-04-19 20:54:52,782 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/A/a262a79346ba4331a4de0393726ca17c as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/a262a79346ba4331a4de0393726ca17c 2023-04-19 20:54:52,785 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 09bc544118dfd67701ac39d6da1a53f4/B of 09bc544118dfd67701ac39d6da1a53f4 into 82eaedbf08294b58830dc03c11c92fd6(size=12.1 K), total size for store is 35.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 20:54:52,785 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 09bc544118dfd67701ac39d6da1a53f4: 2023-04-19 20:54:52,785 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4., storeName=09bc544118dfd67701ac39d6da1a53f4/B, priority=10, startTime=1681937692301; duration=0sec 2023-04-19 20:54:52,785 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=4), splitQueue=0 2023-04-19 20:54:52,785 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 8 store files, 6 compacting, 2 eligible, 16 blocking 2023-04-19 20:54:52,786 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-19 20:54:52,786 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-19 20:54:52,786 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. because compaction request was cancelled 2023-04-19 20:54:52,786 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 8 store files, 0 compacting, 8 eligible, 16 blocking 2023-04-19 20:54:52,794 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 09bc544118dfd67701ac39d6da1a53f4/A of 09bc544118dfd67701ac39d6da1a53f4 into a262a79346ba4331a4de0393726ca17c(size=12.1 K), total size for store is 38.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 20:54:52,794 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 09bc544118dfd67701ac39d6da1a53f4: 2023-04-19 20:54:52,794 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4., storeName=09bc544118dfd67701ac39d6da1a53f4/A, priority=10, startTime=1681937692301; duration=0sec 2023-04-19 20:54:52,795 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-19 20:54:52,795 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 8 files of size 96288 starting at candidate #0 after considering 21 permutations with 21 in ratio 2023-04-19 20:54:52,795 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1912): 09bc544118dfd67701ac39d6da1a53f4/C is initiating minor compaction (all files) 2023-04-19 20:54:52,795 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 8 store files, 8 compacting, 0 eligible, 16 blocking 2023-04-19 20:54:52,795 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-19 20:54:52,795 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-19 20:54:52,795 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 09bc544118dfd67701ac39d6da1a53f4/C in TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. 2023-04-19 20:54:52,795 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. because compaction request was cancelled 2023-04-19 20:54:52,795 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-19 20:54:52,795 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/e8344ec11af845dc877d43c20f89f903, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/99e8bfc6626a4015a341c31dce1ebbb9, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/94abbcc9eebe4d6f9e908f6c71835d25, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/17013f7f119840da8a2a16ea307d7997, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/56b9a028ace1448ca126392a183aa92e, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/7a7a84c1f3a148d18a3938f90ea1a0fd, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/a97c730a972f46ab8a1ee332e28dba03, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/ba02cc56583a4873953ca419c5714aeb] into tmpdir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp, totalSize=94.0 K 2023-04-19 20:54:52,795 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting e8344ec11af845dc877d43c20f89f903, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=10, earliestPutTs=1681937685392 2023-04-19 20:54:52,796 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36764 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-19 20:54:52,796 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1912): 09bc544118dfd67701ac39d6da1a53f4/B is initiating minor compaction (all files) 2023-04-19 20:54:52,796 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 09bc544118dfd67701ac39d6da1a53f4/B in TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. 2023-04-19 20:54:52,796 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 99e8bfc6626a4015a341c31dce1ebbb9, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=64, earliestPutTs=1681937685471 2023-04-19 20:54:52,796 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/82eaedbf08294b58830dc03c11c92fd6, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/bfcf02323c734c7f920ed57c840e7d1e, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/3ea8fc98b3cd432d8fa8c22cc0602a52] into tmpdir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp, totalSize=35.9 K 2023-04-19 20:54:52,796 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 94abbcc9eebe4d6f9e908f6c71835d25, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=73, earliestPutTs=1681937686560 2023-04-19 20:54:52,797 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 82eaedbf08294b58830dc03c11c92fd6, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=277, earliestPutTs=1681937691300 2023-04-19 20:54:52,797 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 17013f7f119840da8a2a16ea307d7997, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=161, earliestPutTs=1681937687760 2023-04-19 20:54:52,797 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting bfcf02323c734c7f920ed57c840e7d1e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=285, earliestPutTs=1681937691452 2023-04-19 20:54:52,797 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 56b9a028ace1448ca126392a183aa92e, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=170, earliestPutTs=1681937688813 2023-04-19 20:54:52,798 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 3ea8fc98b3cd432d8fa8c22cc0602a52, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=296, earliestPutTs=1681937692588 2023-04-19 20:54:52,798 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 7a7a84c1f3a148d18a3938f90ea1a0fd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=277, earliestPutTs=1681937691300 2023-04-19 20:54:52,798 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting a97c730a972f46ab8a1ee332e28dba03, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=285, earliestPutTs=1681937691452 2023-04-19 20:54:52,799 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting ba02cc56583a4873953ca419c5714aeb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=296, earliestPutTs=1681937692588 2023-04-19 20:54:52,803 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:52,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34548 deadline: 1681937752801, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:52,807 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] throttle.PressureAwareThroughputController(145): 09bc544118dfd67701ac39d6da1a53f4#B#compaction#226 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 57.69 MB/second 2023-04-19 20:54:52,823 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] throttle.PressureAwareThroughputController(145): 09bc544118dfd67701ac39d6da1a53f4#C#compaction#227 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 57.69 MB/second 2023-04-19 20:54:52,831 DEBUG [Thread-1624] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:54:52,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:54:52,834 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:54:52,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:54:52,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:54:52,835 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:54:52,835 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:52,835 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:54:52,835 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:54:52,836 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:52,836 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:52,837 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:52,837 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:52,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:52,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:52,838 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:52,838 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:54:52,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:52,838 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:52,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:54:52,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:52,839 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:54:52,839 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:54:52,839 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:54:52,839 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:54:52,839 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:52,839 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:54:52,840 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:54:52,840 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:54:52,840 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. 2023-04-19 20:54:52,840 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. started... 2023-04-19 20:54:52,840 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. as already flushing 2023-04-19 20:54:52,840 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. 2023-04-19 20:54:52,840 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:54:52,840 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:54:52,840 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:52,840 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:54:52,840 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:54:52,843 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:52,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:52,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:52,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:52,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:52,844 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:52,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:52,844 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:52,844 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:54:52,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:52,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:52,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:52,845 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:54:52,845 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@42a0357a[Count = 0] remaining members to acquire global barrier 2023-04-19 20:54:52,845 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:54:52,845 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:52,847 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:52,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:52,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:52,847 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:54:52,847 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:54:52,847 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:54:52,847 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:52,847 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:54:52,848 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:54:52,849 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:52,848 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:52,849 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:54:52,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:52,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:52,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:52,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:52,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:52,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:52,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:52,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:52,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:52,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:52,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:54:52,851 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:54:52,851 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:54:52,851 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:54:52,851 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:54:52,851 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:54:52,851 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:54:52,853 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:52,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:52,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:52,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:52,853 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:52,853 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:52,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:52,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:52,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:52,854 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:52,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:52,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:52,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:52,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:52,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:52,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:52,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:52,855 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:52,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:52,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:52,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:52,858 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:52,858 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:52,858 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:52,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:52,858 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:52,858 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:52,858 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:52,858 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:52,858 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:52,858 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:54:52,858 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:52,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:52,858 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:52,858 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:54:52,859 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:52,859 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:52,859 DEBUG [Thread-1624] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:54:52,859 DEBUG [Thread-1624] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:54:52,906 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:52,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34548 deadline: 1681937752904, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:52,959 DEBUG [Thread-1624] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:54:52,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:54:52,961 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:54:52,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:54:52,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:54:52,962 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:54:52,962 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:52,962 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:54:52,962 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:54:52,963 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:52,963 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:52,964 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:52,964 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:52,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:52,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:52,965 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:52,965 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:54:52,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:52,965 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:52,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:54:52,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:52,966 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:54:52,966 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:54:52,966 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:54:52,966 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:54:52,966 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:52,966 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:54:52,966 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:54:52,966 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:54:52,966 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. 2023-04-19 20:54:52,966 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. started... 2023-04-19 20:54:52,966 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. as already flushing 2023-04-19 20:54:52,966 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. 2023-04-19 20:54:52,967 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:54:52,967 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:54:52,967 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:52,967 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:54:52,967 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:54:52,968 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:52,968 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:52,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:52,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:52,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:52,968 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:52,968 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:54:52,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:52,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:52,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:52,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:52,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:52,970 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:54:52,970 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@33b362a2[Count = 0] remaining members to acquire global barrier 2023-04-19 20:54:52,970 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:54:52,970 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:52,971 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:52,971 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:52,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:52,971 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:54:52,971 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:54:52,971 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:54:52,971 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:52,971 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:54:52,973 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:52,973 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:54:52,973 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:52,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:52,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:52,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:52,973 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:54:52,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:52,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:52,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:52,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:52,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:52,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:52,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:52,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:54:52,976 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:54:52,976 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:54:52,976 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:54:52,976 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:54:52,976 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:54:52,976 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:54:52,977 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:52,977 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:52,978 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:52,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:52,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:52,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:52,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:52,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:52,978 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:52,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:52,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:52,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:52,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:52,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:52,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:52,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:52,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:52,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:52,980 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:52,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:52,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:52,985 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:52,985 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:52,985 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:52,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:52,985 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:52,985 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:52,985 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:52,985 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:52,985 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:54:52,985 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:54:52,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:52,985 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:52,985 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:52,985 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:52,985 DEBUG [Thread-1624] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:54:52,985 DEBUG [Thread-1624] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:54:52,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:52,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:53,086 DEBUG [Thread-1624] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:54:53,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:54:53,088 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:54:53,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:54:53,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:54:53,089 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:54:53,089 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:53,089 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:54:53,089 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:54:53,089 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:53,090 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:53,092 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:53,092 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:53,092 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:53,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:53,093 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:53,093 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:54:53,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:53,094 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:53,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:54:53,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:53,094 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:54:53,094 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:54:53,094 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:54:53,094 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:54:53,094 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:53,095 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:54:53,095 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:54:53,095 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:54:53,095 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. 2023-04-19 20:54:53,095 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. started... 2023-04-19 20:54:53,095 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. as already flushing 2023-04-19 20:54:53,095 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. 2023-04-19 20:54:53,095 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:54:53,095 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:54:53,095 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:53,095 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:54:53,095 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:54:53,096 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:53,096 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:53,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:53,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:53,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:53,097 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:53,097 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:54:53,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:53,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:53,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:53,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:53,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:53,098 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:54:53,098 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:54:53,098 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@243bcc2c[Count = 0] remaining members to acquire global barrier 2023-04-19 20:54:53,098 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:53,100 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:53,100 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:53,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:53,100 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:54:53,100 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:54:53,100 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:54:53,100 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:53,100 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:54:53,101 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:53,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:53,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:53,101 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:54:53,101 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:53,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:53,102 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:54:53,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:53,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:53,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:53,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:53,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:53,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:53,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:53,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:54:53,104 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:54:53,104 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:54:53,104 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:54:53,104 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:54:53,104 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:54:53,104 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:54:53,107 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:53,107 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:53,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:53,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:53,107 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:53,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:53,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:53,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:53,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:53,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:53,108 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:53,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:53,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:53,108 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:53,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34548 deadline: 1681937753107, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:53,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:53,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:53,109 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:53,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:53,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:53,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:53,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:53,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:53,111 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:53,111 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:53,111 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:53,111 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:53,111 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:54:53,111 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:54:53,112 DEBUG [Thread-1624] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:54:53,112 DEBUG [Thread-1624] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:54:53,111 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:53,112 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:53,112 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:53,112 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:53,112 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:53,112 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:53,112 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:54:53,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:53,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:53,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:53,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:53,147 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=310 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/B/47601069633c461a87e550937cb4a47a 2023-04-19 20:54:53,178 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=310 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/C/850542cc5c8b451281c3ade24602a9e8 2023-04-19 20:54:53,185 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/A/5461affcf7804820a05692cd1a68292e as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/5461affcf7804820a05692cd1a68292e 2023-04-19 20:54:53,190 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/5461affcf7804820a05692cd1a68292e, entries=200, sequenceid=310, filesize=14.3 K 2023-04-19 20:54:53,191 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/B/47601069633c461a87e550937cb4a47a as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/47601069633c461a87e550937cb4a47a 2023-04-19 20:54:53,199 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/47601069633c461a87e550937cb4a47a, entries=150, sequenceid=310, filesize=11.9 K 2023-04-19 20:54:53,200 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/C/850542cc5c8b451281c3ade24602a9e8 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/850542cc5c8b451281c3ade24602a9e8 2023-04-19 20:54:53,206 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/850542cc5c8b451281c3ade24602a9e8, entries=150, sequenceid=310, filesize=11.9 K 2023-04-19 20:54:53,207 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 09bc544118dfd67701ac39d6da1a53f4 in 514ms, sequenceid=310, compaction requested=true 2023-04-19 20:54:53,207 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 09bc544118dfd67701ac39d6da1a53f4: 2023-04-19 20:54:53,207 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 20:54:53,207 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-19 20:54:53,207 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=3), splitQueue=0 2023-04-19 20:54:53,212 DEBUG [Thread-1624] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:54:53,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:54:53,214 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:54:53,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:54:53,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:54:53,215 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:54:53,215 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:53,215 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:54:53,215 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:54:53,216 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:53,216 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:53,218 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:53,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:53,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:53,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:53,218 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:53,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:54:53,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:53,219 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:54:53,220 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:54:53,220 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:54:53,220 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:54:53,220 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:53,220 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:54:53,221 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:54:53,221 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:54:53,221 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. 2023-04-19 20:54:53,221 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. started... 2023-04-19 20:54:53,221 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 09bc544118dfd67701ac39d6da1a53f4 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-19 20:54:53,221 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09bc544118dfd67701ac39d6da1a53f4, store=A 2023-04-19 20:54:53,221 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:54:53,221 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09bc544118dfd67701ac39d6da1a53f4, store=B 2023-04-19 20:54:53,221 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:54:53,221 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09bc544118dfd67701ac39d6da1a53f4, store=C 2023-04-19 20:54:53,221 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:54:53,225 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:53,225 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:53,225 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:54:53,230 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/B/ca9c96d6334747d0b3e23833b43f22b2 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/ca9c96d6334747d0b3e23833b43f22b2 2023-04-19 20:54:53,236 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 09bc544118dfd67701ac39d6da1a53f4/B of 09bc544118dfd67701ac39d6da1a53f4 into ca9c96d6334747d0b3e23833b43f22b2(size=12.2 K), total size for store is 24.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 20:54:53,236 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 09bc544118dfd67701ac39d6da1a53f4: 2023-04-19 20:54:53,236 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4., storeName=09bc544118dfd67701ac39d6da1a53f4/B, priority=13, startTime=1681937692691; duration=0sec 2023-04-19 20:54:53,237 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=3), splitQueue=0 2023-04-19 20:54:53,237 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 9 store files, 8 compacting, 1 eligible, 16 blocking 2023-04-19 20:54:53,237 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-19 20:54:53,237 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-19 20:54:53,237 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. because compaction request was cancelled 2023-04-19 20:54:53,237 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-19 20:54:53,239 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 53830 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-19 20:54:53,239 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1912): 09bc544118dfd67701ac39d6da1a53f4/A is initiating minor compaction (all files) 2023-04-19 20:54:53,239 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 09bc544118dfd67701ac39d6da1a53f4/A in TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. 2023-04-19 20:54:53,239 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/a262a79346ba4331a4de0393726ca17c, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/bf46b37dae664e82b75afe78db6c2acb, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/1ba5c75941d74be1b8659e0b8fc6da94, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/5461affcf7804820a05692cd1a68292e] into tmpdir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp, totalSize=52.6 K 2023-04-19 20:54:53,239 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting a262a79346ba4331a4de0393726ca17c, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=277, earliestPutTs=1681937691300 2023-04-19 20:54:53,239 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting bf46b37dae664e82b75afe78db6c2acb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=285, earliestPutTs=1681937691452 2023-04-19 20:54:53,240 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 1ba5c75941d74be1b8659e0b8fc6da94, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=296, earliestPutTs=1681937692588 2023-04-19 20:54:53,240 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 5461affcf7804820a05692cd1a68292e, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=310, earliestPutTs=1681937692661 2023-04-19 20:54:53,249 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/C/7c0d9d7106b94bd58ed78a9afd39c515 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/7c0d9d7106b94bd58ed78a9afd39c515 2023-04-19 20:54:53,254 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1652): Completed compaction of 8 (all) file(s) in 09bc544118dfd67701ac39d6da1a53f4/C of 09bc544118dfd67701ac39d6da1a53f4 into 7c0d9d7106b94bd58ed78a9afd39c515(size=12.2 K), total size for store is 24.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 20:54:53,255 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 09bc544118dfd67701ac39d6da1a53f4: 2023-04-19 20:54:53,255 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4., storeName=09bc544118dfd67701ac39d6da1a53f4/C, priority=8, startTime=1681937692691; duration=0sec 2023-04-19 20:54:53,255 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 20:54:53,255 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-04-19 20:54:53,255 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-19 20:54:53,255 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-19 20:54:53,255 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. because compaction request was cancelled 2023-04-19 20:54:53,257 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] throttle.PressureAwareThroughputController(145): 09bc544118dfd67701ac39d6da1a53f4#A#compaction#230 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 57.69 MB/second 2023-04-19 20:54:53,265 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=330 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/A/c5ad4e0b96d7436d9c57b2c16e44260a 2023-04-19 20:54:53,273 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/A/8cbad9651fdc4c84bedbb94942796c01 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/8cbad9651fdc4c84bedbb94942796c01 2023-04-19 20:54:53,278 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 09bc544118dfd67701ac39d6da1a53f4/A of 09bc544118dfd67701ac39d6da1a53f4 into 8cbad9651fdc4c84bedbb94942796c01(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 20:54:53,278 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 09bc544118dfd67701ac39d6da1a53f4: 2023-04-19 20:54:53,278 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4., storeName=09bc544118dfd67701ac39d6da1a53f4/A, priority=12, startTime=1681937693207; duration=0sec 2023-04-19 20:54:53,278 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 20:54:53,291 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=330 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/B/c962658efe874b22852e9aa20182d5b7 2023-04-19 20:54:53,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(9158): Flush requested on 09bc544118dfd67701ac39d6da1a53f4 2023-04-19 20:54:53,413 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. as already flushing 2023-04-19 20:54:53,438 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:53,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34548 deadline: 1681937753437, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:53,541 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:53,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34548 deadline: 1681937753540, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:53,710 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=330 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/C/a0fc09181c9547ad93ebdc3173541134 2023-04-19 20:54:53,714 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/A/c5ad4e0b96d7436d9c57b2c16e44260a as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/c5ad4e0b96d7436d9c57b2c16e44260a 2023-04-19 20:54:53,718 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/c5ad4e0b96d7436d9c57b2c16e44260a, entries=150, sequenceid=330, filesize=11.9 K 2023-04-19 20:54:53,719 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/B/c962658efe874b22852e9aa20182d5b7 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/c962658efe874b22852e9aa20182d5b7 2023-04-19 20:54:53,723 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/c962658efe874b22852e9aa20182d5b7, entries=150, sequenceid=330, filesize=11.9 K 2023-04-19 20:54:53,724 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/C/a0fc09181c9547ad93ebdc3173541134 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/a0fc09181c9547ad93ebdc3173541134 2023-04-19 20:54:53,727 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/a0fc09181c9547ad93ebdc3173541134, entries=150, sequenceid=330, filesize=11.9 K 2023-04-19 20:54:53,728 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 09bc544118dfd67701ac39d6da1a53f4 in 507ms, sequenceid=330, compaction requested=true 2023-04-19 20:54:53,728 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 09bc544118dfd67701ac39d6da1a53f4: 2023-04-19 20:54:53,728 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. 2023-04-19 20:54:53,728 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:54:53,728 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:54:53,728 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:53,728 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:54:53,728 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:54:53,729 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:53,730 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:53,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:53,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:53,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:53,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:53,730 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:53,730 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:54:53,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:53,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:53,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:53,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:53,731 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:54:53,731 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7940fa6a[Count = 0] remaining members to acquire global barrier 2023-04-19 20:54:53,731 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:54:53,731 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:53,733 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:53,733 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:53,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:53,733 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:54:53,734 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:54:53,734 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:54:53,734 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:53,734 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:54:53,735 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:53,735 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:54:53,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:53,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:53,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:53,735 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:53,735 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:54:53,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:53,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:53,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:53,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:53,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:53,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:53,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:53,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:54:53,738 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:54:53,738 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:54:53,738 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:54:53,738 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:54:53,738 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:54:53,738 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:54:53,740 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:53,740 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:53,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:53,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:53,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:53,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:53,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:53,740 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:53,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:53,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:53,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:53,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:53,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:53,741 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:53,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:53,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:53,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:53,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:53,742 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:53,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:53,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:53,744 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:53,744 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:53,744 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:53,745 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:53,745 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:53,745 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:54:53,745 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:54:53,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:53,745 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:53,745 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:53,745 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:53,745 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:53,745 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:53,745 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:53,745 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:53,745 DEBUG [Thread-1624] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:54:53,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:53,745 DEBUG [Thread-1624] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:54:53,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(9158): Flush requested on 09bc544118dfd67701ac39d6da1a53f4 2023-04-19 20:54:53,746 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 09bc544118dfd67701ac39d6da1a53f4 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-19 20:54:53,746 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09bc544118dfd67701ac39d6da1a53f4, store=A 2023-04-19 20:54:53,746 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:54:53,746 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09bc544118dfd67701ac39d6da1a53f4, store=B 2023-04-19 20:54:53,746 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:54:53,746 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09bc544118dfd67701ac39d6da1a53f4, store=C 2023-04-19 20:54:53,746 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:54:53,789 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:53,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 243 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34548 deadline: 1681937753787, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:53,845 DEBUG [Thread-1624] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:54:53,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:54:53,847 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:54:53,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:54:53,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:54:53,849 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:54:53,849 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:53,849 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:54:53,849 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:54:53,849 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:53,849 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:53,851 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:53,851 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:53,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:53,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:53,852 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:53,852 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:54:53,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:53,852 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:53,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:54:53,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:53,852 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:54:53,853 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:54:53,853 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:54:53,853 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:54:53,853 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:53,853 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:54:53,853 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:54:53,853 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:54:53,853 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. 2023-04-19 20:54:53,854 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. started... 2023-04-19 20:54:53,854 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. as already flushing 2023-04-19 20:54:53,854 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. 2023-04-19 20:54:53,854 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:54:53,854 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:54:53,854 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:53,854 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:54:53,854 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:54:53,856 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:53,856 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:53,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:53,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:53,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:53,856 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:53,856 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:54:53,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:53,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:53,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:53,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:53,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:53,857 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:54:53,857 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3a080bdd[Count = 0] remaining members to acquire global barrier 2023-04-19 20:54:53,857 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:54:53,857 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:53,858 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:53,858 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:53,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:53,859 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:54:53,859 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:54:53,859 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:54:53,859 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:53,859 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:54:53,860 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:53,860 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:54:53,860 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:53,860 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:54:53,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:53,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:53,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:53,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:53,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:53,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:53,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:53,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:53,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:53,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:53,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:54:53,863 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:54:53,863 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:54:53,863 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:54:53,863 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:54:53,863 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:54:53,863 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:54:53,865 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:53,865 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:53,866 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:53,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:53,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:53,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:53,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:53,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:53,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:53,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:53,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:53,866 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:53,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:53,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:53,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:53,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:53,867 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:53,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:53,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:53,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:53,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:53,869 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:53,869 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:53,869 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:53,869 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:53,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:53,869 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:53,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:53,870 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:53,870 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:54:53,870 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:54:53,869 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:53,870 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:53,870 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:53,870 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:53,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:53,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:53,870 DEBUG [Thread-1624] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:54:53,870 DEBUG [Thread-1624] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:54:53,870 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:54:53,891 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:53,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 245 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34548 deadline: 1681937753890, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:53,970 DEBUG [Thread-1624] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:54:53,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:54:53,972 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:54:53,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:54:53,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:54:53,973 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:54:53,973 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:53,973 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:54:53,973 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:54:53,974 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:53,974 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:53,981 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:53,981 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:53,981 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:53,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:53,981 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:53,981 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:54:53,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:53,982 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:53,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:54:53,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:53,982 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:54:53,982 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:54:53,982 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:54:53,982 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:54:53,982 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:53,982 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:54:53,983 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:54:53,983 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:54:53,983 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. 2023-04-19 20:54:53,983 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. started... 2023-04-19 20:54:53,983 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. as already flushing 2023-04-19 20:54:53,983 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. 2023-04-19 20:54:53,983 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:54:53,983 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:54:53,983 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:53,983 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:54:53,983 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:54:53,987 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:53,987 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:53,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:53,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:53,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:53,988 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:53,988 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:54:53,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:53,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:53,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:53,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:53,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:53,989 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:54:53,989 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1206c84c[Count = 0] remaining members to acquire global barrier 2023-04-19 20:54:53,989 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:54:53,989 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:53,990 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:53,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:53,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:53,990 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:54:53,990 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:53,991 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:54:53,990 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:54:53,991 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:54:53,992 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:54:53,992 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:53,992 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:53,992 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:54:53,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:53,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:53,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:53,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:53,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:53,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:53,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:53,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:53,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:53,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:53,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:54:53,995 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:54:53,995 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:54:53,995 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:54:53,995 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:54:53,995 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:54:53,995 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:54:53,998 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:53,998 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:53,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:53,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:53,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:53,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:53,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:53,998 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:53,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:53,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:53,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:53,998 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:53,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:53,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:53,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:53,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:54,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:54,000 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:54,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:54,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:54,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:54,002 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:54,002 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:54,002 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:54,002 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:54,002 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:54,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:54,002 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:54,002 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:54:54,002 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:54:54,002 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:54,002 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:54,002 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:54,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:54,003 DEBUG [Thread-1624] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:54:54,003 DEBUG [Thread-1624] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:54:54,003 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:54,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:54,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:54,095 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:54,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 247 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34548 deadline: 1681937754093, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:54,103 DEBUG [Thread-1624] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:54:54,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:54:54,105 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:54:54,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:54:54,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:54:54,106 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:54:54,106 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:54,106 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:54:54,106 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:54:54,107 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:54,107 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:54,108 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:54,108 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:54,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:54,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:54,109 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:54,109 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:54:54,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:54,109 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:54,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:54:54,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:54,109 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:54:54,110 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:54:54,110 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:54:54,110 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:54:54,110 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:54,110 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:54:54,110 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:54:54,110 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:54:54,111 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. 2023-04-19 20:54:54,111 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. started... 2023-04-19 20:54:54,111 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. as already flushing 2023-04-19 20:54:54,111 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. 2023-04-19 20:54:54,111 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:54:54,111 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:54:54,111 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:54,111 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:54:54,111 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:54:54,112 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:54,112 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:54,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:54,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:54,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:54,113 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:54,113 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:54:54,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:54,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:54,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:54,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:54,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:54,114 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:54:54,114 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@68263c8c[Count = 0] remaining members to acquire global barrier 2023-04-19 20:54:54,114 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:54:54,114 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:54,117 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:54,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:54,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:54,117 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:54:54,117 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:54,117 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:54:54,117 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:54:54,117 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:54:54,119 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:54,119 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:54:54,119 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:54,119 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:54:54,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:54,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:54,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:54,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:54,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:54,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:54,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:54,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:54,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:54,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:54,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:54:54,122 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:54:54,122 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:54:54,122 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:54:54,122 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:54:54,122 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:54:54,122 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:54:54,123 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:54,123 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:54,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:54,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:54,123 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:54,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:54,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:54,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:54,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:54,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:54,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:54,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:54,124 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:54,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:54,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:54,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:54,125 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:54,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:54,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:54,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:54,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:54,128 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:54,128 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:54,128 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:54,128 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:54,128 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:54,128 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:54,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:54,128 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:54,128 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:54:54,128 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:54:54,128 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:54,128 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:54,128 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:54,128 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:54,128 DEBUG [Thread-1624] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:54:54,128 DEBUG [Thread-1624] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:54:54,128 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:54,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:54,154 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=348 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/A/80a13ac695664bdc9436659c7fda85f7 2023-04-19 20:54:54,229 DEBUG [Thread-1624] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:54:54,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:54:54,231 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:54:54,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:54:54,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:54:54,232 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:54:54,232 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:54,232 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:54:54,232 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:54:54,232 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:54,232 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:54,235 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:54,235 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:54,235 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:54,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:54,236 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:54,236 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:54:54,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:54,236 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:54,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:54:54,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:54,237 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:54:54,237 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:54:54,237 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:54:54,237 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:54:54,237 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:54,237 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:54:54,237 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:54:54,237 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:54:54,237 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. 2023-04-19 20:54:54,237 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. started... 2023-04-19 20:54:54,237 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. as already flushing 2023-04-19 20:54:54,237 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. 2023-04-19 20:54:54,238 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:54:54,238 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:54:54,238 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:54,238 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:54:54,238 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:54:54,239 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:54,239 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:54,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:54,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:54,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:54,240 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:54,240 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:54:54,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:54,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:54,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:54,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:54,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:54,241 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:54:54,241 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@164b0848[Count = 0] remaining members to acquire global barrier 2023-04-19 20:54:54,241 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:54:54,241 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:54,242 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:54,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:54,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:54,243 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:54:54,243 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:54:54,243 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:54:54,243 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:54,243 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:54:54,244 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:54:54,244 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:54,244 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:54,244 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:54:54,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:54,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:54,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:54,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:54,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:54,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:54,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:54,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:54,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:54,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:54,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:54:54,246 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:54:54,246 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:54:54,247 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:54:54,247 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:54:54,247 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:54:54,247 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:54:54,250 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:54,250 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:54,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:54,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:54,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:54,250 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:54,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:54,250 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:54,250 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:54,250 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:54,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:54,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:54,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:54,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:54,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:54,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:54,251 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:54,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:54,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:54,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:54,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:54,254 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:54,254 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:54,254 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:54,254 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:54,254 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:54,254 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:54,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:54,254 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:54:54,254 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:54:54,254 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:54,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:54,254 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:54,254 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:54,254 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:54:54,254 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:54,254 DEBUG [Thread-1624] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:54:54,254 DEBUG [Thread-1624] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:54:54,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:54,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:54,354 DEBUG [Thread-1624] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:54:54,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:54:54,356 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:54:54,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:54:54,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:54:54,358 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:54:54,358 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:54,358 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:54:54,358 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:54:54,358 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:54,358 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:54,360 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:54,360 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:54,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:54,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:54,360 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:54,360 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:54:54,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:54,361 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:54,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:54:54,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:54,361 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:54:54,361 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:54:54,361 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:54:54,361 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:54:54,361 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:54,361 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:54:54,361 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:54:54,362 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:54:54,362 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. 2023-04-19 20:54:54,362 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. started... 2023-04-19 20:54:54,362 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. as already flushing 2023-04-19 20:54:54,362 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. 2023-04-19 20:54:54,362 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:54:54,362 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:54:54,362 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:54,362 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:54:54,362 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:54:54,363 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:54,363 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:54,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:54,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:54,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:54,364 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:54,364 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:54:54,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:54,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:54,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:54,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:54,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:54,365 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:54:54,365 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:54:54,365 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ce7afbf[Count = 0] remaining members to acquire global barrier 2023-04-19 20:54:54,365 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:54,367 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:54,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:54,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:54,367 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:54:54,367 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:54,367 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:54:54,367 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:54:54,367 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:54:54,369 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:54,369 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:54:54,369 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:54,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:54,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:54,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:54,369 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:54:54,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:54,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:54,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:54,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:54,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:54,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:54,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:54,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:54:54,371 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:54:54,371 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:54:54,371 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:54:54,371 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:54:54,371 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:54:54,371 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:54:54,373 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:54,373 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:54,373 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:54,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:54,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:54,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:54,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:54,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:54,374 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:54,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:54,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:54,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:54,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:54,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:54,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:54,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:54,375 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:54,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:54,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:54,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:54,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:54,377 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:54,377 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:54,377 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:54,377 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:54,377 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:54,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:54,377 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:54,377 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:54:54,377 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:54:54,377 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:54:54,377 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:54,378 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:54,378 DEBUG [Thread-1624] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:54:54,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:54,378 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:54,378 DEBUG [Thread-1624] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:54:54,378 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:54,378 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:54,378 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:54,400 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:54,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 249 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34548 deadline: 1681937754398, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:54,478 DEBUG [Thread-1624] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:54:54,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:54:54,480 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:54:54,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:54:54,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:54:54,481 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:54:54,481 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:54,481 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:54:54,481 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:54:54,482 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:54,482 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:54,484 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:54,484 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:54,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:54,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:54,484 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:54,484 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:54:54,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:54,484 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:54,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:54:54,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:54,485 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:54:54,485 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:54:54,485 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:54:54,485 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:54:54,485 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:54,485 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:54:54,485 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:54:54,485 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:54:54,485 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. 2023-04-19 20:54:54,485 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. started... 2023-04-19 20:54:54,486 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. as already flushing 2023-04-19 20:54:54,486 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. 2023-04-19 20:54:54,486 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:54:54,486 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:54:54,486 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:54,486 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:54:54,486 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:54:54,487 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:54,487 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:54,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:54,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:54,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:54,488 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:54,488 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:54:54,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:54,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:54,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:54,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:54,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:54,489 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:54:54,489 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@48874ec7[Count = 0] remaining members to acquire global barrier 2023-04-19 20:54:54,489 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:54:54,489 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:54,490 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:54,490 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:54,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:54,491 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:54:54,491 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:54:54,491 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:54:54,491 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:54,491 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:54:54,492 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:54:54,492 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:54,492 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:54,492 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:54:54,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:54,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:54,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:54,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:54,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:54,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:54,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:54,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:54,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:54,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:54,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:54:54,495 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:54:54,495 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:54:54,495 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:54:54,495 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:54:54,495 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:54:54,495 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:54:54,500 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:54,500 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:54,500 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:54,500 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:54,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:54,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:54,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:54,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:54,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:54,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:54,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:54,501 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:54,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:54,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:54,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:54,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:54,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:54,502 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:54,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:54,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:54,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:54,504 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:54,505 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:54,505 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:54,505 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:54,505 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:54,505 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:54:54,505 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:54:54,505 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:54,505 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:54,505 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:54,505 DEBUG [Thread-1624] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:54:54,505 DEBUG [Thread-1624] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:54:54,505 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:54:54,505 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:54,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:54,505 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:54,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:54,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:54,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:54,564 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=348 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/B/26f2bb18a4f14358b03957a3f5ade3fe 2023-04-19 20:54:54,574 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=348 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/C/3faa2fe5bd2f4bf6934b5ea37774aaf5 2023-04-19 20:54:54,577 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/A/80a13ac695664bdc9436659c7fda85f7 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/80a13ac695664bdc9436659c7fda85f7 2023-04-19 20:54:54,581 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/80a13ac695664bdc9436659c7fda85f7, entries=150, sequenceid=348, filesize=11.9 K 2023-04-19 20:54:54,581 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/B/26f2bb18a4f14358b03957a3f5ade3fe as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/26f2bb18a4f14358b03957a3f5ade3fe 2023-04-19 20:54:54,584 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/26f2bb18a4f14358b03957a3f5ade3fe, entries=150, sequenceid=348, filesize=11.9 K 2023-04-19 20:54:54,585 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/C/3faa2fe5bd2f4bf6934b5ea37774aaf5 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/3faa2fe5bd2f4bf6934b5ea37774aaf5 2023-04-19 20:54:54,588 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/3faa2fe5bd2f4bf6934b5ea37774aaf5, entries=150, sequenceid=348, filesize=11.9 K 2023-04-19 20:54:54,589 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 09bc544118dfd67701ac39d6da1a53f4 in 843ms, sequenceid=348, compaction requested=true 2023-04-19 20:54:54,589 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 09bc544118dfd67701ac39d6da1a53f4: 2023-04-19 20:54:54,589 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 20:54:54,589 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-19 20:54:54,589 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-19 20:54:54,589 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 20:54:54,590 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 20:54:54,590 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36900 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-19 20:54:54,590 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1912): 09bc544118dfd67701ac39d6da1a53f4/A is initiating minor compaction (all files) 2023-04-19 20:54:54,590 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 09bc544118dfd67701ac39d6da1a53f4/A in TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. 2023-04-19 20:54:54,590 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/8cbad9651fdc4c84bedbb94942796c01, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/c5ad4e0b96d7436d9c57b2c16e44260a, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/80a13ac695664bdc9436659c7fda85f7] into tmpdir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp, totalSize=36.0 K 2023-04-19 20:54:54,591 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49052 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-19 20:54:54,591 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1912): 09bc544118dfd67701ac39d6da1a53f4/B is initiating minor compaction (all files) 2023-04-19 20:54:54,591 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 8cbad9651fdc4c84bedbb94942796c01, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=310, earliestPutTs=1681937692673 2023-04-19 20:54:54,591 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 09bc544118dfd67701ac39d6da1a53f4/B in TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. 2023-04-19 20:54:54,591 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/ca9c96d6334747d0b3e23833b43f22b2, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/47601069633c461a87e550937cb4a47a, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/c962658efe874b22852e9aa20182d5b7, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/26f2bb18a4f14358b03957a3f5ade3fe] into tmpdir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp, totalSize=47.9 K 2023-04-19 20:54:54,591 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting c5ad4e0b96d7436d9c57b2c16e44260a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=330, earliestPutTs=1681937692750 2023-04-19 20:54:54,591 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting ca9c96d6334747d0b3e23833b43f22b2, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=296, earliestPutTs=1681937692588 2023-04-19 20:54:54,591 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 80a13ac695664bdc9436659c7fda85f7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=348, earliestPutTs=1681937693433 2023-04-19 20:54:54,591 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 47601069633c461a87e550937cb4a47a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=310, earliestPutTs=1681937692673 2023-04-19 20:54:54,592 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting c962658efe874b22852e9aa20182d5b7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=330, earliestPutTs=1681937692750 2023-04-19 20:54:54,592 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 26f2bb18a4f14358b03957a3f5ade3fe, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=348, earliestPutTs=1681937693433 2023-04-19 20:54:54,598 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] throttle.PressureAwareThroughputController(145): 09bc544118dfd67701ac39d6da1a53f4#A#compaction#236 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 57.69 MB/second 2023-04-19 20:54:54,600 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] throttle.PressureAwareThroughputController(145): 09bc544118dfd67701ac39d6da1a53f4#B#compaction#237 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 57.69 MB/second 2023-04-19 20:54:54,605 DEBUG [Thread-1624] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:54:54,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:54:54,608 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:54:54,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:54:54,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:54:54,609 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:54:54,609 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:54,609 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:54:54,609 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:54:54,610 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:54,610 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:54,611 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:54,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:54,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:54,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:54,611 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:54,612 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:54,612 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:54,612 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:54:54,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:54:54,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:54,613 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:54:54,613 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:54:54,613 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:54:54,613 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:54:54,613 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:54,613 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/B/38592d5aee7b47ad8d8413ea030a6c8f as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/38592d5aee7b47ad8d8413ea030a6c8f 2023-04-19 20:54:54,613 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:54:54,613 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:54:54,613 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:54:54,613 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. 2023-04-19 20:54:54,613 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. started... 2023-04-19 20:54:54,614 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 09bc544118dfd67701ac39d6da1a53f4 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-19 20:54:54,614 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09bc544118dfd67701ac39d6da1a53f4, store=A 2023-04-19 20:54:54,614 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:54:54,615 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09bc544118dfd67701ac39d6da1a53f4, store=B 2023-04-19 20:54:54,615 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:54:54,615 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09bc544118dfd67701ac39d6da1a53f4, store=C 2023-04-19 20:54:54,615 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:54:54,618 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 09bc544118dfd67701ac39d6da1a53f4/B of 09bc544118dfd67701ac39d6da1a53f4 into 38592d5aee7b47ad8d8413ea030a6c8f(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 20:54:54,618 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 09bc544118dfd67701ac39d6da1a53f4: 2023-04-19 20:54:54,618 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4., storeName=09bc544118dfd67701ac39d6da1a53f4/B, priority=12, startTime=1681937694589; duration=0sec 2023-04-19 20:54:54,618 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 20:54:54,618 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-19 20:54:54,620 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49018 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-19 20:54:54,620 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1912): 09bc544118dfd67701ac39d6da1a53f4/C is initiating minor compaction (all files) 2023-04-19 20:54:54,620 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 09bc544118dfd67701ac39d6da1a53f4/C in TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. 2023-04-19 20:54:54,620 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/7c0d9d7106b94bd58ed78a9afd39c515, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/850542cc5c8b451281c3ade24602a9e8, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/a0fc09181c9547ad93ebdc3173541134, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/3faa2fe5bd2f4bf6934b5ea37774aaf5] into tmpdir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp, totalSize=47.9 K 2023-04-19 20:54:54,620 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 7c0d9d7106b94bd58ed78a9afd39c515, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=296, earliestPutTs=1681937692588 2023-04-19 20:54:54,621 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 850542cc5c8b451281c3ade24602a9e8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=310, earliestPutTs=1681937692673 2023-04-19 20:54:54,621 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting a0fc09181c9547ad93ebdc3173541134, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=330, earliestPutTs=1681937692750 2023-04-19 20:54:54,621 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 3faa2fe5bd2f4bf6934b5ea37774aaf5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=348, earliestPutTs=1681937693433 2023-04-19 20:54:54,629 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] throttle.PressureAwareThroughputController(145): 09bc544118dfd67701ac39d6da1a53f4#C#compaction#239 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 57.69 MB/second 2023-04-19 20:54:54,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(9158): Flush requested on 09bc544118dfd67701ac39d6da1a53f4 2023-04-19 20:54:54,904 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. as already flushing 2023-04-19 20:54:54,958 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:54,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 263 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34548 deadline: 1681937754954, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:55,011 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/A/fa86b1b7451848e2b27bd1f6a10d8015 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/fa86b1b7451848e2b27bd1f6a10d8015 2023-04-19 20:54:55,016 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 09bc544118dfd67701ac39d6da1a53f4/A of 09bc544118dfd67701ac39d6da1a53f4 into fa86b1b7451848e2b27bd1f6a10d8015(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 20:54:55,016 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 09bc544118dfd67701ac39d6da1a53f4: 2023-04-19 20:54:55,016 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4., storeName=09bc544118dfd67701ac39d6da1a53f4/A, priority=13, startTime=1681937694589; duration=0sec 2023-04-19 20:54:55,016 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 20:54:55,026 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=365 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/A/8fa5b2d10c1e4bbba458b1a74e069e2b 2023-04-19 20:54:55,037 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/C/4490115a7b484a308c8ee5e9b419118a as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/4490115a7b484a308c8ee5e9b419118a 2023-04-19 20:54:55,041 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 09bc544118dfd67701ac39d6da1a53f4/C of 09bc544118dfd67701ac39d6da1a53f4 into 4490115a7b484a308c8ee5e9b419118a(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 20:54:55,041 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 09bc544118dfd67701ac39d6da1a53f4: 2023-04-19 20:54:55,041 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4., storeName=09bc544118dfd67701ac39d6da1a53f4/C, priority=12, startTime=1681937694590; duration=0sec 2023-04-19 20:54:55,041 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 20:54:55,060 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:55,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 265 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34548 deadline: 1681937755059, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:55,263 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:55,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 267 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34548 deadline: 1681937755262, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:55,437 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=365 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/B/076ea9fbd3224c9e9f9142dc8f61a8ee 2023-04-19 20:54:55,566 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:55,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 269 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34548 deadline: 1681937755565, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:55,847 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=365 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/C/74506f098f154c4b8d5b42249e4a9a0a 2023-04-19 20:54:55,852 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/A/8fa5b2d10c1e4bbba458b1a74e069e2b as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/8fa5b2d10c1e4bbba458b1a74e069e2b 2023-04-19 20:54:55,856 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/8fa5b2d10c1e4bbba458b1a74e069e2b, entries=150, sequenceid=365, filesize=11.9 K 2023-04-19 20:54:55,856 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/B/076ea9fbd3224c9e9f9142dc8f61a8ee as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/076ea9fbd3224c9e9f9142dc8f61a8ee 2023-04-19 20:54:55,861 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/076ea9fbd3224c9e9f9142dc8f61a8ee, entries=150, sequenceid=365, filesize=11.9 K 2023-04-19 20:54:55,862 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/C/74506f098f154c4b8d5b42249e4a9a0a as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/74506f098f154c4b8d5b42249e4a9a0a 2023-04-19 20:54:55,867 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/74506f098f154c4b8d5b42249e4a9a0a, entries=150, sequenceid=365, filesize=11.9 K 2023-04-19 20:54:55,867 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 09bc544118dfd67701ac39d6da1a53f4 in 1254ms, sequenceid=365, compaction requested=false 2023-04-19 20:54:55,867 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 09bc544118dfd67701ac39d6da1a53f4: 2023-04-19 20:54:55,868 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. 2023-04-19 20:54:55,868 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:54:55,868 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:54:55,868 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:55,868 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:54:55,868 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:54:55,869 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:55,869 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:55,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:55,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:55,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:55,870 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:55,870 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:54:55,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:55,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:55,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:55,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:55,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:55,871 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:54:55,871 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@111cd907[Count = 0] remaining members to acquire global barrier 2023-04-19 20:54:55,871 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:54:55,871 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:55,874 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:55,874 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:55,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:55,874 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:54:55,874 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:54:55,874 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:54:55,874 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:55,874 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:54:55,875 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:55,875 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:54:55,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:55,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:55,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:55,876 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:55,876 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:54:55,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:55,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:55,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:55,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:55,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:55,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:55,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:55,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(9158): Flush requested on 09bc544118dfd67701ac39d6da1a53f4 2023-04-19 20:54:55,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:54:55,878 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 09bc544118dfd67701ac39d6da1a53f4 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-19 20:54:55,878 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:54:55,878 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:54:55,878 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:54:55,878 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:54:55,878 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:54:55,878 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:54:55,878 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09bc544118dfd67701ac39d6da1a53f4, store=A 2023-04-19 20:54:55,879 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:54:55,879 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09bc544118dfd67701ac39d6da1a53f4, store=B 2023-04-19 20:54:55,879 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:54:55,879 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09bc544118dfd67701ac39d6da1a53f4, store=C 2023-04-19 20:54:55,879 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:54:55,880 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:55,880 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:55,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:55,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:55,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:55,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:55,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:55,880 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:55,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:55,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:55,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:55,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:55,881 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:55,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:55,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:55,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:55,882 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:55,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:55,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:55,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:55,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:55,885 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:55,885 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:55,885 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:55,885 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:55,885 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:54:55,885 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:54:55,885 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:55,885 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:55,885 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:55,885 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:55,885 DEBUG [Thread-1624] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:54:55,885 DEBUG [Thread-1624] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:54:55,885 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:54:55,885 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:55,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:55,885 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:55,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:55,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:55,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:55,913 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:55,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34540 deadline: 1681937755910, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:55,913 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:55,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34552 deadline: 1681937755911, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:55,915 DEBUG [Thread-1616] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8148 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4., hostname=jenkins-hbase4.apache.org,35247,1681937616971, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-19 20:54:55,916 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:55,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34572 deadline: 1681937755913, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:55,935 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:55,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34562 deadline: 1681937755932, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:55,937 DEBUG [Thread-1622] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8170 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4., hostname=jenkins-hbase4.apache.org,35247,1681937616971, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-19 20:54:55,985 DEBUG [Thread-1624] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:54:55,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:54:55,988 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:54:55,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:54:55,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:54:55,989 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:54:55,989 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:55,989 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:54:55,989 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:54:55,990 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:55,990 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:55,992 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:55,992 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:55,992 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:55,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:55,992 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:55,992 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:54:55,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:55,992 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:55,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:54:55,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:55,993 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:54:55,993 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:54:55,993 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:54:55,993 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:54:55,993 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:55,993 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:54:55,993 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:54:55,994 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:54:55,994 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. 2023-04-19 20:54:55,994 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. started... 2023-04-19 20:54:55,994 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. as already flushing 2023-04-19 20:54:55,994 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. 2023-04-19 20:54:55,994 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:54:55,994 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:54:55,994 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:55,994 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:54:55,994 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:54:55,995 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:55,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:55,995 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:55,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:55,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:55,996 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:55,996 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:54:55,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:55,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:55,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:55,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:55,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:55,998 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:54:55,998 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5fdb9a24[Count = 0] remaining members to acquire global barrier 2023-04-19 20:54:55,998 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:54:55,998 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:55,999 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:55,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:55,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:55,999 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:54:55,999 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:54:55,999 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:54:55,999 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:55,999 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:54:56,003 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:56,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:56,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:56,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:56,004 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:54:56,004 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:56,004 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:54:56,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:56,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:56,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:56,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:56,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:56,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:56,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:56,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:54:56,008 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:54:56,008 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:54:56,008 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:54:56,008 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:54:56,008 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:54:56,008 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:54:56,010 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:56,010 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:56,010 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:56,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:56,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:56,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:56,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:56,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:56,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:56,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:56,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:56,011 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:56,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:56,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:56,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:56,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:56,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:56,012 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:56,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:56,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:56,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:56,015 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:56,015 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:56,015 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:54:56,015 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:54:56,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34540 deadline: 1681937756014, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:56,015 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:56,015 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:56,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:56,016 DEBUG [Thread-1624] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:54:56,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:56,016 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:56,016 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:56,016 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:56,016 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:56,016 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:56,016 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:56,016 DEBUG [Thread-1624] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:54:56,016 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:56,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:56,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:56,018 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:56,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34572 deadline: 1681937756017, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:56,069 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:56,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 271 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34548 deadline: 1681937756068, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:56,116 DEBUG [Thread-1624] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:54:56,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:54:56,118 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:54:56,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:54:56,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:54:56,120 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:54:56,120 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:56,120 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:54:56,120 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:54:56,121 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:56,121 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:56,122 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:56,122 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:56,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:56,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:56,122 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:56,122 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:54:56,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:56,123 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:56,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:54:56,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:56,123 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:54:56,123 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:54:56,123 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:54:56,124 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:54:56,124 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:56,124 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:54:56,124 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:54:56,124 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:54:56,124 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. 2023-04-19 20:54:56,124 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. started... 2023-04-19 20:54:56,124 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. as already flushing 2023-04-19 20:54:56,124 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. 2023-04-19 20:54:56,124 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:54:56,124 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:54:56,124 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:56,124 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:54:56,124 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:54:56,127 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:56,127 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:56,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:56,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:56,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:56,127 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:56,127 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:54:56,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:56,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:56,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:56,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:56,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:56,129 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:54:56,129 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@e104c37[Count = 0] remaining members to acquire global barrier 2023-04-19 20:54:56,129 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:54:56,129 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:56,130 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:56,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:56,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:56,130 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:54:56,130 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:54:56,130 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:54:56,130 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:56,130 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:54:56,131 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:54:56,131 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:56,131 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:56,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:56,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:56,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:56,131 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:54:56,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:56,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:56,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:56,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:56,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:56,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:56,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:56,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:54:56,134 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:54:56,134 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:54:56,134 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:54:56,134 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:54:56,134 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:54:56,134 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:54:56,136 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:56,136 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:56,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:56,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:56,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:56,136 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:56,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:56,136 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:56,136 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:56,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:56,136 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:56,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:56,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:56,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:56,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:56,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:56,138 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:56,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:56,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:56,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:56,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:56,141 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:56,141 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:56,141 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:56,141 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:56,141 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:54:56,141 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:56,141 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:54:56,141 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:54:56,141 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:56,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:56,141 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:56,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:56,141 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:56,141 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:56,141 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:56,141 DEBUG [Thread-1624] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:54:56,141 DEBUG [Thread-1624] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:54:56,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:56,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:56,217 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:56,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34540 deadline: 1681937756216, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:56,220 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:56,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34572 deadline: 1681937756219, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:56,242 DEBUG [Thread-1624] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:54:56,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:54:56,244 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:54:56,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:54:56,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:54:56,245 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:54:56,245 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:56,245 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:54:56,245 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:54:56,246 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:56,246 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:56,247 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:56,247 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:56,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:56,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:56,247 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:56,247 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:54:56,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:56,248 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:56,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:54:56,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:56,248 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:54:56,248 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:54:56,248 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:54:56,248 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:54:56,248 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:56,248 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:54:56,249 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:54:56,249 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:54:56,249 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. 2023-04-19 20:54:56,249 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. started... 2023-04-19 20:54:56,249 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. as already flushing 2023-04-19 20:54:56,249 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. 2023-04-19 20:54:56,249 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:54:56,249 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:54:56,249 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:56,249 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:54:56,249 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:54:56,250 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:56,250 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:56,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:56,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:56,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:56,251 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:56,251 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:54:56,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:56,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:56,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:56,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:56,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:56,253 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:54:56,253 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7c93da44[Count = 0] remaining members to acquire global barrier 2023-04-19 20:54:56,253 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:54:56,253 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:56,254 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:56,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:56,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:56,255 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:54:56,255 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:54:56,255 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:54:56,255 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:56,255 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:54:56,257 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:56,257 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:54:56,257 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:56,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:56,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:56,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:56,257 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:54:56,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:56,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:56,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:56,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:56,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:56,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:56,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:56,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:54:56,259 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:54:56,259 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:54:56,259 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:54:56,259 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:54:56,259 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:54:56,259 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:54:56,261 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:56,261 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:56,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:56,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:56,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:56,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:56,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:56,261 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:56,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:56,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:56,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:56,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:56,262 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:56,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:56,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:56,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:56,263 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:56,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:56,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:56,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:56,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:56,266 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:56,266 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:56,266 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:56,266 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:56,266 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:56,266 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:54:56,266 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:54:56,266 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:56,266 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:54:56,266 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:56,266 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:56,266 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:56,266 DEBUG [Thread-1624] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:54:56,266 DEBUG [Thread-1624] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:54:56,266 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:56,266 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:56,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:56,267 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:56,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:56,290 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=384 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/A/7dbaba8ccae443c0a92a61d498d00efd 2023-04-19 20:54:56,366 DEBUG [Thread-1624] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:54:56,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:54:56,368 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:54:56,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:54:56,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:54:56,371 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:54:56,371 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:56,371 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:54:56,371 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:54:56,372 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:56,372 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:56,373 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:56,373 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:56,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:56,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:56,373 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:56,373 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:54:56,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:56,374 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:56,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:54:56,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:56,374 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:54:56,374 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:54:56,374 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:54:56,374 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:54:56,375 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:56,375 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:54:56,375 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:54:56,375 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:54:56,375 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. 2023-04-19 20:54:56,375 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. started... 2023-04-19 20:54:56,375 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. as already flushing 2023-04-19 20:54:56,375 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. 2023-04-19 20:54:56,375 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:54:56,375 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:54:56,375 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:56,375 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:54:56,375 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:54:56,378 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:56,378 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:56,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:56,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:56,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:56,378 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:56,378 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:54:56,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:56,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:56,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:56,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:56,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:56,379 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:54:56,379 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@bb0f030[Count = 0] remaining members to acquire global barrier 2023-04-19 20:54:56,379 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:54:56,380 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:56,381 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:56,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:56,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:56,381 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:54:56,381 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:54:56,381 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:54:56,381 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:56,381 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:54:56,382 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:56,382 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:54:56,382 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:56,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:56,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:56,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:56,382 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:54:56,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:56,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:56,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:56,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:56,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:56,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:56,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:56,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:54:56,385 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:54:56,385 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:54:56,385 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:54:56,385 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:54:56,385 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:54:56,385 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:54:56,386 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:56,386 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:56,386 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:56,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:56,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:56,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:56,386 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:56,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:56,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:56,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:56,387 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:56,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:56,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:56,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:56,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:56,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:56,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:56,389 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:56,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:56,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:56,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:56,394 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:56,394 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:56,394 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:56,394 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:56,394 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:56,394 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:54:56,394 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:54:56,394 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:56,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:56,394 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:56,395 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:56,395 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:56,395 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:56,395 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:56,395 DEBUG [Thread-1624] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:54:56,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:56,395 DEBUG [Thread-1624] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:54:56,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:56,495 DEBUG [Thread-1624] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:54:56,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:54:56,497 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:54:56,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:54:56,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:54:56,498 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:54:56,498 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:56,498 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:54:56,498 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:54:56,499 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:56,499 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:56,500 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:56,500 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:56,500 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:56,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:56,500 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:56,500 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:54:56,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:56,500 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:56,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:54:56,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:56,501 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:54:56,501 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:54:56,501 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:54:56,501 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:54:56,501 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:56,501 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:54:56,501 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:54:56,501 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:54:56,501 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. 2023-04-19 20:54:56,501 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. started... 2023-04-19 20:54:56,501 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. as already flushing 2023-04-19 20:54:56,501 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. 2023-04-19 20:54:56,502 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:54:56,502 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:54:56,502 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:56,502 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:54:56,502 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:54:56,503 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:56,503 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:56,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:56,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:56,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:56,503 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:56,503 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:54:56,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:56,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:56,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:56,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:56,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:56,505 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:54:56,505 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@8b941d8[Count = 0] remaining members to acquire global barrier 2023-04-19 20:54:56,505 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:54:56,505 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:56,506 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:56,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:56,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:56,506 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:54:56,506 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:54:56,506 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:54:56,506 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:56,506 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:54:56,509 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:56,509 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:54:56,509 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:56,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:56,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:56,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:56,509 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:54:56,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:56,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:56,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:56,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:56,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:56,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:56,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:56,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:54:56,511 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:54:56,511 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:54:56,511 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:54:56,511 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:54:56,511 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:54:56,511 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:54:56,513 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:56,513 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:56,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:56,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:56,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:56,513 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:56,513 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:56,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:56,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:56,513 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:56,513 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:56,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:56,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:56,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:56,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:56,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:56,514 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:56,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:56,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:56,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:56,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:56,517 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:56,517 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:56,517 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:56,517 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:56,517 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:56,517 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:56,517 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:56,517 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:54:56,517 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:54:56,517 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:56,517 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:56,517 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:54:56,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:56,517 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:56,517 DEBUG [Thread-1624] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:54:56,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:56,517 DEBUG [Thread-1624] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:54:56,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:56,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:56,521 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:56,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34540 deadline: 1681937756519, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:56,523 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:56,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34572 deadline: 1681937756522, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:56,618 DEBUG [Thread-1624] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:54:56,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:54:56,620 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:54:56,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:54:56,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:54:56,621 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:54:56,621 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:56,621 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:54:56,621 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:54:56,621 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:56,621 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:56,623 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:56,623 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:56,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:56,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:56,623 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:56,623 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:54:56,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:56,623 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:56,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:54:56,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:56,624 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:54:56,624 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:54:56,625 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:54:56,625 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:54:56,625 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:56,625 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:54:56,625 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:54:56,625 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:54:56,625 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. 2023-04-19 20:54:56,625 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. started... 2023-04-19 20:54:56,625 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. as already flushing 2023-04-19 20:54:56,625 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. 2023-04-19 20:54:56,625 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:54:56,625 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:54:56,625 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:56,625 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:54:56,625 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:54:56,628 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:56,628 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:56,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:56,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:56,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:56,628 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:56,628 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:54:56,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:56,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:56,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:56,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:56,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:56,630 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:54:56,630 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4c4c50c[Count = 0] remaining members to acquire global barrier 2023-04-19 20:54:56,630 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:54:56,630 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:56,631 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:56,631 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:56,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:56,631 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:54:56,631 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:56,631 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:54:56,631 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:54:56,632 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:54:56,633 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:56,633 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:54:56,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:56,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:56,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:56,633 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:56,633 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:54:56,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:56,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:56,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:56,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:56,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:56,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:56,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:56,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:54:56,635 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:54:56,635 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:54:56,635 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:54:56,635 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:54:56,635 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:54:56,635 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:54:56,637 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:56,637 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:56,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:56,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:56,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:56,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:56,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:56,637 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:56,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:56,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:56,637 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:56,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:56,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:56,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:56,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:56,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:56,638 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:56,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:56,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:56,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:56,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:56,642 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:56,643 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:56,643 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:56,643 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:56,643 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:56,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:56,643 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:56,643 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:54:56,643 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:56,643 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:56,643 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:56,643 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:54:56,643 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:54:56,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:56,643 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:56,643 DEBUG [Thread-1624] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:54:56,643 DEBUG [Thread-1624] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:54:56,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:56,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:56,700 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=384 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/B/3e0baffb4b364763b046329b1769b9d6 2023-04-19 20:54:56,743 DEBUG [Thread-1624] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:54:56,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:54:56,746 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:54:56,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:54:56,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:54:56,747 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:54:56,747 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:56,747 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:54:56,747 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:54:56,747 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:56,747 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:56,749 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:56,749 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:56,749 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:56,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:56,749 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:56,749 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:54:56,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:56,750 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:56,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:54:56,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:56,750 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:54:56,750 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:54:56,750 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:54:56,750 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:54:56,750 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:56,750 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:54:56,751 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:54:56,751 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:54:56,751 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. 2023-04-19 20:54:56,751 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. started... 2023-04-19 20:54:56,751 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. as already flushing 2023-04-19 20:54:56,751 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. 2023-04-19 20:54:56,751 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:54:56,751 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:54:56,751 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:56,751 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:54:56,751 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:54:56,753 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:56,753 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:56,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:56,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:56,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:56,753 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:56,753 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:54:56,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:56,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:56,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:56,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:56,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:56,754 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:54:56,754 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@66e77e2a[Count = 0] remaining members to acquire global barrier 2023-04-19 20:54:56,754 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:54:56,754 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:56,755 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:56,756 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:56,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:56,756 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:54:56,756 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:56,756 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:54:56,756 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:54:56,756 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:54:56,758 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:56,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:56,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:56,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:56,758 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:54:56,759 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:56,759 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:54:56,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:56,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:56,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:56,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:56,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:56,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:56,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:56,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:54:56,760 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:54:56,760 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:54:56,760 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:54:56,761 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:54:56,761 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:54:56,761 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:54:56,762 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:56,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:56,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:56,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:56,762 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:56,762 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:56,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:56,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:56,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:56,762 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:56,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:56,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:56,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:56,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:56,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:56,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:56,764 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:56,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:56,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:56,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:56,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:56,769 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:56,769 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:56,769 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:56,769 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:56,769 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:56,769 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:56,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:56,769 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:56,769 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:54:56,769 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:54:56,769 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:54:56,769 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:56,769 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:56,769 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:56,769 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:56,769 DEBUG [Thread-1624] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:54:56,769 DEBUG [Thread-1624] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:54:56,769 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:56,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:56,869 DEBUG [Thread-1624] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:54:56,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:54:56,872 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:54:56,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:54:56,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:54:56,873 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:54:56,873 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:56,873 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:54:56,873 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:54:56,873 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:56,873 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:56,876 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:56,876 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:56,876 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:56,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:56,877 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:56,877 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:54:56,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:56,877 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:56,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:54:56,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:56,877 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:54:56,878 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:54:56,878 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:54:56,878 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:54:56,878 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:56,878 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:54:56,878 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:54:56,878 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:54:56,878 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. 2023-04-19 20:54:56,878 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. started... 2023-04-19 20:54:56,878 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. as already flushing 2023-04-19 20:54:56,878 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. 2023-04-19 20:54:56,878 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:54:56,878 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:54:56,878 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:56,878 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:54:56,878 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:54:56,881 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:56,881 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:56,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:56,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:56,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:56,882 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:56,882 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:54:56,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:56,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:56,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:56,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:56,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:56,883 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:54:56,883 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a5a33b6[Count = 0] remaining members to acquire global barrier 2023-04-19 20:54:56,883 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:54:56,883 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:56,886 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:56,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:56,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:56,886 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:54:56,886 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:54:56,886 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:56,886 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:54:56,886 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:54:56,889 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:56,889 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:54:56,889 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:56,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:56,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:56,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:56,890 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:54:56,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:56,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:56,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:56,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:56,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:56,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:56,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:56,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:54:56,892 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:54:56,892 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:54:56,892 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:54:56,892 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:54:56,892 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:54:56,892 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:54:56,895 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:56,895 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:56,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:56,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:56,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:56,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:56,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:56,895 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:56,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:56,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:56,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:56,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:56,896 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:56,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:56,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:56,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:56,897 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:56,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:56,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:56,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:56,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:56,902 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:56,902 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:56,902 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:56,902 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:56,902 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:54:56,902 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:54:56,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:56,902 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:54:56,902 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:56,902 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:56,902 DEBUG [Thread-1624] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:54:56,902 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:56,902 DEBUG [Thread-1624] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:54:56,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:56,902 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:56,903 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:56,903 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:56,903 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:56,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:57,003 DEBUG [Thread-1624] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:54:57,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:54:57,005 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:54:57,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:54:57,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:54:57,006 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:54:57,006 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:57,006 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:54:57,006 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:54:57,007 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:57,007 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:57,008 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:57,008 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:57,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:57,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:57,008 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:57,009 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:54:57,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:57,009 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:57,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:54:57,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:57,009 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:54:57,009 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:54:57,010 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:54:57,010 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:54:57,010 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:57,010 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:54:57,010 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:54:57,010 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:54:57,010 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. 2023-04-19 20:54:57,010 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. started... 2023-04-19 20:54:57,010 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. as already flushing 2023-04-19 20:54:57,010 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. 2023-04-19 20:54:57,010 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:54:57,010 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:54:57,010 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:57,010 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:54:57,010 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:54:57,012 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:57,012 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:57,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:57,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:57,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:57,012 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:57,012 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:54:57,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:57,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:57,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:57,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:57,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:57,014 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:54:57,014 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:54:57,014 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@282f18a4[Count = 0] remaining members to acquire global barrier 2023-04-19 20:54:57,014 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:57,015 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:57,015 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:57,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:57,016 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:54:57,016 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:54:57,016 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:54:57,016 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:57,016 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:54:57,017 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:57,017 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:54:57,017 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:57,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:57,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:57,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:57,017 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:54:57,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:57,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:57,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:57,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:57,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:57,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:57,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:57,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:54:57,019 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:54:57,019 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:54:57,019 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:54:57,019 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:54:57,019 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:54:57,019 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:54:57,021 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:57,021 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:57,021 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:57,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:57,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:57,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:57,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:57,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:57,022 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:57,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:57,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:57,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:57,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:57,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:57,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:57,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:57,023 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:57,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:57,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:57,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:57,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:57,025 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:57,025 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:57,025 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:57,025 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:57,025 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:57,025 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:54:57,025 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:54:57,025 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:57,025 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:57,025 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:57,025 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:54:57,025 DEBUG [Thread-1624] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:54:57,025 DEBUG [Thread-1624] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:54:57,026 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:57,026 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:57,026 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:57,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:57,026 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:57,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:57,026 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:57,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34540 deadline: 1681937757025, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:57,028 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:57,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34572 deadline: 1681937757027, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:57,074 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:57,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 273 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34548 deadline: 1681937757073, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:57,110 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=384 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/C/7d59300d429f48e5b6a630e2a0c20dba 2023-04-19 20:54:57,114 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/A/7dbaba8ccae443c0a92a61d498d00efd as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/7dbaba8ccae443c0a92a61d498d00efd 2023-04-19 20:54:57,118 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/7dbaba8ccae443c0a92a61d498d00efd, entries=200, sequenceid=384, filesize=14.3 K 2023-04-19 20:54:57,118 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/B/3e0baffb4b364763b046329b1769b9d6 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/3e0baffb4b364763b046329b1769b9d6 2023-04-19 20:54:57,121 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/3e0baffb4b364763b046329b1769b9d6, entries=150, sequenceid=384, filesize=11.9 K 2023-04-19 20:54:57,122 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/C/7d59300d429f48e5b6a630e2a0c20dba as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/7d59300d429f48e5b6a630e2a0c20dba 2023-04-19 20:54:57,125 DEBUG [Thread-1624] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:54:57,125 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/7d59300d429f48e5b6a630e2a0c20dba, entries=150, sequenceid=384, filesize=11.9 K 2023-04-19 20:54:57,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:54:57,126 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 09bc544118dfd67701ac39d6da1a53f4 in 1248ms, sequenceid=384, compaction requested=true 2023-04-19 20:54:57,126 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 09bc544118dfd67701ac39d6da1a53f4: 2023-04-19 20:54:57,126 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 20:54:57,126 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-19 20:54:57,127 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-19 20:54:57,126 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 20:54:57,127 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 20:54:57,127 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39442 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-19 20:54:57,128 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1912): 09bc544118dfd67701ac39d6da1a53f4/A is initiating minor compaction (all files) 2023-04-19 20:54:57,128 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37002 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-19 20:54:57,128 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 09bc544118dfd67701ac39d6da1a53f4/A in TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. 2023-04-19 20:54:57,128 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/fa86b1b7451848e2b27bd1f6a10d8015, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/8fa5b2d10c1e4bbba458b1a74e069e2b, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/7dbaba8ccae443c0a92a61d498d00efd] into tmpdir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp, totalSize=38.5 K 2023-04-19 20:54:57,128 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:54:57,128 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1912): 09bc544118dfd67701ac39d6da1a53f4/B is initiating minor compaction (all files) 2023-04-19 20:54:57,128 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 09bc544118dfd67701ac39d6da1a53f4/B in TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. 2023-04-19 20:54:57,128 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting fa86b1b7451848e2b27bd1f6a10d8015, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=348, earliestPutTs=1681937693433 2023-04-19 20:54:57,128 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/38592d5aee7b47ad8d8413ea030a6c8f, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/076ea9fbd3224c9e9f9142dc8f61a8ee, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/3e0baffb4b364763b046329b1769b9d6] into tmpdir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp, totalSize=36.1 K 2023-04-19 20:54:57,129 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 38592d5aee7b47ad8d8413ea030a6c8f, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=348, earliestPutTs=1681937693433 2023-04-19 20:54:57,129 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 8fa5b2d10c1e4bbba458b1a74e069e2b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=365, earliestPutTs=1681937693776 2023-04-19 20:54:57,129 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 076ea9fbd3224c9e9f9142dc8f61a8ee, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=365, earliestPutTs=1681937693776 2023-04-19 20:54:57,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:54:57,129 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 7dbaba8ccae443c0a92a61d498d00efd, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=384, earliestPutTs=1681937694933 2023-04-19 20:54:57,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:54:57,129 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:54:57,129 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:57,129 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:54:57,129 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 3e0baffb4b364763b046329b1769b9d6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=384, earliestPutTs=1681937694950 2023-04-19 20:54:57,129 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:54:57,130 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:57,130 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:57,132 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:57,132 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:57,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:57,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:57,133 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:57,133 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:54:57,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:57,133 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:57,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:54:57,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:57,134 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:54:57,144 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:54:57,144 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:54:57,144 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:54:57,144 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:57,144 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:54:57,145 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] throttle.PressureAwareThroughputController(145): 09bc544118dfd67701ac39d6da1a53f4#B#compaction#245 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 57.69 MB/second 2023-04-19 20:54:57,146 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:54:57,146 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:54:57,146 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. 2023-04-19 20:54:57,146 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. started... 2023-04-19 20:54:57,146 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] throttle.PressureAwareThroughputController(145): 09bc544118dfd67701ac39d6da1a53f4#A#compaction#246 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 57.69 MB/second 2023-04-19 20:54:57,146 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 09bc544118dfd67701ac39d6da1a53f4 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-19 20:54:57,146 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09bc544118dfd67701ac39d6da1a53f4, store=A 2023-04-19 20:54:57,147 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:54:57,147 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09bc544118dfd67701ac39d6da1a53f4, store=B 2023-04-19 20:54:57,147 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:54:57,147 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09bc544118dfd67701ac39d6da1a53f4, store=C 2023-04-19 20:54:57,147 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:54:57,157 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/B/8fb851af21914f5793e7128f1612a95b as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/8fb851af21914f5793e7128f1612a95b 2023-04-19 20:54:57,159 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=400 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/A/dff48212e2c2468b9a91ef1ac8a770d0 2023-04-19 20:54:57,161 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/A/fdd7f4955e3349349d002125a8a6eeec as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/fdd7f4955e3349349d002125a8a6eeec 2023-04-19 20:54:57,163 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 09bc544118dfd67701ac39d6da1a53f4/B of 09bc544118dfd67701ac39d6da1a53f4 into 8fb851af21914f5793e7128f1612a95b(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 20:54:57,163 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 09bc544118dfd67701ac39d6da1a53f4: 2023-04-19 20:54:57,163 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4., storeName=09bc544118dfd67701ac39d6da1a53f4/B, priority=13, startTime=1681937697126; duration=0sec 2023-04-19 20:54:57,163 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 20:54:57,163 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-19 20:54:57,165 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36968 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-19 20:54:57,165 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1912): 09bc544118dfd67701ac39d6da1a53f4/C is initiating minor compaction (all files) 2023-04-19 20:54:57,165 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 09bc544118dfd67701ac39d6da1a53f4/C in TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. 2023-04-19 20:54:57,165 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/4490115a7b484a308c8ee5e9b419118a, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/74506f098f154c4b8d5b42249e4a9a0a, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/7d59300d429f48e5b6a630e2a0c20dba] into tmpdir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp, totalSize=36.1 K 2023-04-19 20:54:57,166 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 4490115a7b484a308c8ee5e9b419118a, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=348, earliestPutTs=1681937693433 2023-04-19 20:54:57,166 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 74506f098f154c4b8d5b42249e4a9a0a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=365, earliestPutTs=1681937693776 2023-04-19 20:54:57,166 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 7d59300d429f48e5b6a630e2a0c20dba, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=384, earliestPutTs=1681937694950 2023-04-19 20:54:57,168 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 09bc544118dfd67701ac39d6da1a53f4/A of 09bc544118dfd67701ac39d6da1a53f4 into fdd7f4955e3349349d002125a8a6eeec(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 20:54:57,168 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 09bc544118dfd67701ac39d6da1a53f4: 2023-04-19 20:54:57,168 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4., storeName=09bc544118dfd67701ac39d6da1a53f4/A, priority=13, startTime=1681937697126; duration=0sec 2023-04-19 20:54:57,168 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 20:54:57,178 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] throttle.PressureAwareThroughputController(145): 09bc544118dfd67701ac39d6da1a53f4#C#compaction#249 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 57.69 MB/second 2023-04-19 20:54:57,188 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/C/082774a643da40d09836c6b732c57cfb as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/082774a643da40d09836c6b732c57cfb 2023-04-19 20:54:57,192 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 09bc544118dfd67701ac39d6da1a53f4/C of 09bc544118dfd67701ac39d6da1a53f4 into 082774a643da40d09836c6b732c57cfb(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 20:54:57,192 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 09bc544118dfd67701ac39d6da1a53f4: 2023-04-19 20:54:57,192 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4., storeName=09bc544118dfd67701ac39d6da1a53f4/C, priority=13, startTime=1681937697127; duration=0sec 2023-04-19 20:54:57,192 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 20:54:57,583 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=400 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/B/4893a404865c4f69aa7f8dbfb8bd3a08 2023-04-19 20:54:57,993 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=400 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/C/f1b48a2afa87427883ae753f87ddcfa6 2023-04-19 20:54:57,997 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/A/dff48212e2c2468b9a91ef1ac8a770d0 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/dff48212e2c2468b9a91ef1ac8a770d0 2023-04-19 20:54:58,000 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/dff48212e2c2468b9a91ef1ac8a770d0, entries=150, sequenceid=400, filesize=11.9 K 2023-04-19 20:54:58,001 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/B/4893a404865c4f69aa7f8dbfb8bd3a08 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/4893a404865c4f69aa7f8dbfb8bd3a08 2023-04-19 20:54:58,005 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/4893a404865c4f69aa7f8dbfb8bd3a08, entries=150, sequenceid=400, filesize=11.9 K 2023-04-19 20:54:58,006 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/C/f1b48a2afa87427883ae753f87ddcfa6 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/f1b48a2afa87427883ae753f87ddcfa6 2023-04-19 20:54:58,009 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/f1b48a2afa87427883ae753f87ddcfa6, entries=150, sequenceid=400, filesize=11.9 K 2023-04-19 20:54:58,010 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=0 B/0 for 09bc544118dfd67701ac39d6da1a53f4 in 864ms, sequenceid=400, compaction requested=false 2023-04-19 20:54:58,010 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 09bc544118dfd67701ac39d6da1a53f4: 2023-04-19 20:54:58,010 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. 2023-04-19 20:54:58,010 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:54:58,010 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:54:58,010 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:58,010 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:54:58,010 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:54:58,012 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:58,012 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:58,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:58,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:58,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:58,012 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:58,012 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:54:58,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:58,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:58,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:58,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:58,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:58,014 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:54:58,014 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@ae15408[Count = 0] remaining members to acquire global barrier 2023-04-19 20:54:58,014 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:54:58,014 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:58,015 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:58,015 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:58,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:58,015 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:54:58,015 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:54:58,015 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:54:58,015 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:58,016 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:54:58,017 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:54:58,017 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:58,017 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:58,017 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:54:58,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:58,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:58,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:58,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:58,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:58,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:58,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:58,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:58,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:58,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:58,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:54:58,019 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:54:58,019 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:54:58,019 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:54:58,019 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:54:58,019 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:54:58,019 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:54:58,022 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:58,022 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:58,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:58,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:58,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:58,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:58,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:58,022 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:58,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:58,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:58,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:58,023 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:58,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:58,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:58,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:58,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:58,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:58,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:58,025 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:58,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:58,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:58,027 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:58,027 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:58,027 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:58,027 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:58,027 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:54:58,027 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:54:58,027 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:58,027 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:58,027 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:58,027 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:58,027 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:58,027 DEBUG [Thread-1624] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:54:58,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:58,027 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:58,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:58,027 DEBUG [Thread-1624] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:54:58,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:58,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:58,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(9158): Flush requested on 09bc544118dfd67701ac39d6da1a53f4 2023-04-19 20:54:58,042 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 09bc544118dfd67701ac39d6da1a53f4 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-19 20:54:58,042 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09bc544118dfd67701ac39d6da1a53f4, store=A 2023-04-19 20:54:58,042 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:54:58,042 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09bc544118dfd67701ac39d6da1a53f4, store=B 2023-04-19 20:54:58,042 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:54:58,043 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09bc544118dfd67701ac39d6da1a53f4, store=C 2023-04-19 20:54:58,043 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:54:58,051 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=413 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/A/6bb51a26606649e7b436816cc9f29277 2023-04-19 20:54:58,069 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681937698062, max timestamp=1681937698067Num uniques 859; 2023-04-19 20:54:58,069 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681937698062, max timestamp=1681937698067Num uniques 859; 2023-04-19 20:54:58,069 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 20:54:58,069 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 20:54:58,069 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681937698062, max timestamp=1681937698067Num uniques 859; 2023-04-19 20:54:58,069 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 20:54:58,096 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:58,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34572 deadline: 1681937758094, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:58,098 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:58,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34540 deadline: 1681937758095, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:58,128 DEBUG [Thread-1624] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:54:58,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:54:58,130 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:54:58,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:54:58,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:54:58,131 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:54:58,131 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:58,131 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:54:58,131 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:54:58,131 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:58,131 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:58,135 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:58,135 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:58,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:58,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:58,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:58,135 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:58,135 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:54:58,136 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:58,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:54:58,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:58,136 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:54:58,136 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:54:58,136 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:54:58,136 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:54:58,136 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:58,136 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:54:58,136 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:54:58,136 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:54:58,136 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. 2023-04-19 20:54:58,137 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. started... 2023-04-19 20:54:58,137 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. as already flushing 2023-04-19 20:54:58,137 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. 2023-04-19 20:54:58,137 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:54:58,137 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:54:58,137 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:58,137 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:54:58,137 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:54:58,139 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:58,139 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:58,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:58,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:58,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:58,139 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:58,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:58,139 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:54:58,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:58,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:58,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:58,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:58,141 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:54:58,141 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@57aa6d50[Count = 0] remaining members to acquire global barrier 2023-04-19 20:54:58,141 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:54:58,141 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:58,142 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:58,142 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:58,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:58,143 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:54:58,143 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:54:58,143 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:54:58,143 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:58,143 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:54:58,144 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:54:58,144 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:58,144 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:58,144 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:54:58,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:58,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:58,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:58,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:58,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:58,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:58,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:58,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:58,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:58,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:58,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:54:58,146 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:54:58,146 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:54:58,146 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:54:58,146 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:54:58,146 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:54:58,146 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:54:58,148 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:58,148 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:58,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:58,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:58,148 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:58,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:58,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:58,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:58,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:58,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:58,148 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:58,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:58,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:58,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:58,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:58,150 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:58,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:58,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:58,152 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:58,152 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:58,152 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:58,152 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:58,152 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:58,152 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:58,152 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:54:58,153 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:54:58,152 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:58,152 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:58,153 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:58,153 DEBUG [Thread-1624] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:54:58,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:58,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:58,153 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:58,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:58,153 DEBUG [Thread-1624] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:54:58,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:58,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:58,153 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:54:58,198 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:58,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34572 deadline: 1681937758197, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:58,200 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:58,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34540 deadline: 1681937758199, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:58,253 DEBUG [Thread-1624] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:54:58,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:54:58,255 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:54:58,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:54:58,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:54:58,257 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:54:58,257 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:58,257 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:54:58,257 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:54:58,257 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:58,257 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:58,258 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:58,258 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:58,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:58,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:58,259 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:58,259 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:54:58,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:58,259 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:58,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:54:58,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:58,260 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:54:58,260 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:54:58,260 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:54:58,260 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:54:58,260 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:58,260 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:54:58,260 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:54:58,260 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:54:58,260 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. 2023-04-19 20:54:58,260 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. started... 2023-04-19 20:54:58,260 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. as already flushing 2023-04-19 20:54:58,260 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. 2023-04-19 20:54:58,260 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:54:58,260 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:54:58,261 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:58,261 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:54:58,261 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:54:58,262 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:58,262 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:58,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:58,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:58,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:58,262 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:58,262 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:54:58,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:58,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:58,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:58,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:58,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:58,264 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:54:58,264 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6c562390[Count = 0] remaining members to acquire global barrier 2023-04-19 20:54:58,264 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:54:58,264 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:58,265 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:58,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:58,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:58,265 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:54:58,265 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:54:58,265 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:54:58,265 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:58,265 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:54:58,267 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:58,267 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:54:58,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:58,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:58,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:58,267 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:58,267 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:54:58,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:58,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:58,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:58,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:58,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:58,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:58,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:58,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:54:58,269 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:54:58,269 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:54:58,269 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:54:58,269 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:54:58,269 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:54:58,269 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:54:58,271 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:58,271 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:58,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:58,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:58,271 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:58,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:58,271 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:58,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:58,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:58,272 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:58,272 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:58,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:58,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:58,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:58,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:58,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:58,273 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:58,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:58,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:58,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:58,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:58,275 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:58,275 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:58,275 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:58,275 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:54:58,275 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:54:58,275 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:58,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:58,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:58,276 DEBUG [Thread-1624] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:54:58,276 DEBUG [Thread-1624] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:54:58,275 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:58,275 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:58,276 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:54:58,276 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:58,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:58,276 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:58,276 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:58,276 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:58,276 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:58,376 DEBUG [Thread-1624] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:54:58,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:54:58,378 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:54:58,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:54:58,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:54:58,379 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:54:58,379 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:58,379 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:54:58,379 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:54:58,379 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:58,380 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:58,381 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:58,381 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:58,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:58,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:58,381 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:58,381 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:54:58,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:58,381 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:58,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:54:58,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:58,382 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:54:58,382 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:54:58,382 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:54:58,382 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:54:58,382 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:58,382 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:54:58,382 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:54:58,382 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:54:58,383 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. 2023-04-19 20:54:58,383 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. started... 2023-04-19 20:54:58,383 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. as already flushing 2023-04-19 20:54:58,383 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. 2023-04-19 20:54:58,383 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:54:58,383 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:54:58,383 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:58,383 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:54:58,383 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:54:58,385 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:58,385 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:58,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:58,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:58,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:58,386 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:58,386 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:54:58,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:58,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:58,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:58,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:58,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:58,387 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:54:58,387 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3cecfc86[Count = 0] remaining members to acquire global barrier 2023-04-19 20:54:58,387 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:54:58,387 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:58,388 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:58,388 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:58,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:58,388 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:54:58,388 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:54:58,388 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:54:58,388 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:58,389 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:54:58,390 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:58,390 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:54:58,390 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:58,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:58,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:58,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:58,390 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:54:58,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:58,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:58,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:58,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:58,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:58,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:58,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:58,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:54:58,392 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:54:58,392 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:54:58,392 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:54:58,392 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:54:58,392 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:54:58,392 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:54:58,394 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:58,394 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:58,394 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:58,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:58,394 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:58,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:58,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:58,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:58,394 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:58,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:58,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:58,394 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:58,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:58,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:58,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:58,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:58,396 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:58,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:58,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:58,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:58,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:58,399 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:58,399 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:58,399 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:58,399 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:58,399 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:58,400 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:58,400 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:58,400 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:58,400 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:58,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:58,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:58,400 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:54:58,400 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:54:58,400 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:58,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:58,400 DEBUG [Thread-1624] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:54:58,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:58,400 DEBUG [Thread-1624] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:54:58,403 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:58,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34572 deadline: 1681937758401, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:58,403 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:58,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34540 deadline: 1681937758402, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:58,460 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=413 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/B/e1801e9ceb804253841b6fdc70000bcc 2023-04-19 20:54:58,500 DEBUG [Thread-1624] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:54:58,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:54:58,502 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:54:58,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:54:58,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:54:58,504 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:54:58,504 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:58,504 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:54:58,504 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:54:58,504 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:58,504 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:58,506 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:58,506 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:58,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:58,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:58,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:58,507 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:58,507 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:54:58,507 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:58,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:54:58,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:58,507 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:54:58,508 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:54:58,508 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:54:58,508 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:54:58,508 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:58,508 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:54:58,508 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:54:58,508 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:54:58,508 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. 2023-04-19 20:54:58,508 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. started... 2023-04-19 20:54:58,508 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. as already flushing 2023-04-19 20:54:58,508 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. 2023-04-19 20:54:58,508 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:54:58,508 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:54:58,508 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:58,508 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:54:58,508 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:54:58,512 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:58,512 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:58,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:58,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:58,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:58,512 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:58,513 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:54:58,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:58,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:58,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:58,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:58,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:58,514 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:54:58,514 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3cadae07[Count = 0] remaining members to acquire global barrier 2023-04-19 20:54:58,514 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:54:58,514 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:58,515 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:58,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:58,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:58,516 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:54:58,516 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:54:58,516 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:58,516 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:54:58,516 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:54:58,517 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:54:58,517 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:58,517 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:58,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:58,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:58,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:58,517 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:54:58,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:58,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:58,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:58,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:58,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:58,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:58,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:58,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:54:58,520 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:54:58,520 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:54:58,520 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:54:58,520 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:54:58,520 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:54:58,520 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:54:58,521 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:58,521 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:58,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:58,522 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:58,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:58,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:58,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:58,522 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:58,522 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:58,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:58,522 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:58,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:58,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:58,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:58,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:58,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:58,523 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:58,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:58,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:58,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:58,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:58,525 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:58,525 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:58,525 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:58,525 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:58,525 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:58,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:58,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:58,525 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:54:58,526 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:54:58,525 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:58,525 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:58,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:58,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:58,526 DEBUG [Thread-1624] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:54:58,526 DEBUG [Thread-1624] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:54:58,526 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:54:58,526 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:58,526 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:58,526 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:58,626 DEBUG [Thread-1624] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:54:58,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:54:58,628 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:54:58,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:54:58,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:54:58,629 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:54:58,629 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:58,630 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:54:58,630 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:54:58,630 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:58,630 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:58,633 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:58,633 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:58,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:58,633 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:58,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:58,633 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:58,633 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:54:58,634 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:58,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:54:58,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:58,634 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:54:58,634 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:54:58,634 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:54:58,634 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:54:58,634 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:58,634 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:54:58,635 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:54:58,635 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:54:58,635 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. 2023-04-19 20:54:58,635 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. started... 2023-04-19 20:54:58,635 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. as already flushing 2023-04-19 20:54:58,635 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. 2023-04-19 20:54:58,635 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:54:58,635 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:54:58,635 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:58,635 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:54:58,635 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:54:58,637 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:58,637 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:58,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:58,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:58,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:58,637 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:58,637 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:54:58,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:58,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:58,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:58,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:58,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:58,639 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:54:58,639 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3869842f[Count = 0] remaining members to acquire global barrier 2023-04-19 20:54:58,639 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:54:58,639 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:58,640 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:58,640 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:58,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:58,640 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:54:58,640 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:54:58,640 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:58,640 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:54:58,640 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:54:58,641 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:54:58,641 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:58,641 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:58,642 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:54:58,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:58,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:58,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:58,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:58,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:58,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:58,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:58,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:58,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:58,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:58,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:54:58,644 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:54:58,644 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:54:58,645 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:54:58,645 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:54:58,645 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:54:58,645 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:54:58,647 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:58,647 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:58,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:58,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:58,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:58,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:58,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:58,647 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:58,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:58,648 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:58,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:58,648 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:58,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:58,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:58,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:58,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:58,649 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:58,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:58,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:58,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:58,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:58,651 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:58,651 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:58,651 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:58,651 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:58,651 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:54:58,651 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:54:58,651 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:54:58,651 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:58,652 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:58,652 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:58,652 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:58,651 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:58,652 DEBUG [Thread-1624] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:54:58,651 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:58,651 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:58,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:58,652 DEBUG [Thread-1624] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:54:58,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:58,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:58,706 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:58,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34572 deadline: 1681937758705, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:58,706 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:58,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34540 deadline: 1681937758706, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:58,752 DEBUG [Thread-1624] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:54:58,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:54:58,754 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:54:58,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:54:58,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:54:58,755 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:54:58,755 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:58,755 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:54:58,755 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:54:58,756 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:58,756 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:58,757 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:58,757 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:58,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:58,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:58,757 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:58,757 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:54:58,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:58,758 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:58,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:54:58,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:58,758 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:54:58,758 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:54:58,759 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:54:58,759 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:54:58,759 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:58,759 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:54:58,759 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:54:58,759 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:54:58,759 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. 2023-04-19 20:54:58,759 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. started... 2023-04-19 20:54:58,759 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. as already flushing 2023-04-19 20:54:58,759 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. 2023-04-19 20:54:58,759 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:54:58,759 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:54:58,759 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:58,759 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:54:58,759 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:54:58,761 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:58,761 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:58,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:58,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:58,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:58,761 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:58,761 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:54:58,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:58,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:58,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:58,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:58,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:58,762 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:54:58,762 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:54:58,762 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@65fe6a11[Count = 0] remaining members to acquire global barrier 2023-04-19 20:54:58,762 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:58,767 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:58,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:58,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:58,767 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:54:58,767 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:58,767 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:54:58,767 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:54:58,767 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:54:58,769 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:58,769 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:54:58,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:58,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:58,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:58,769 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:58,769 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:54:58,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:58,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:58,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:58,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:58,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:58,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:58,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:58,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:54:58,771 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:54:58,771 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:54:58,772 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:54:58,772 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:54:58,772 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:54:58,772 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:54:58,773 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:58,773 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:58,773 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:58,773 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:58,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:58,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:58,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:58,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:58,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:58,774 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:58,774 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:58,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:58,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:58,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:58,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:58,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:58,775 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:58,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:58,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:58,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:58,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:58,777 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:58,777 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:58,777 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:58,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:58,777 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:54:58,778 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:54:58,777 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:58,777 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:58,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:58,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:58,778 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:58,778 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:58,778 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:58,778 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:58,778 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:58,778 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:58,778 DEBUG [Thread-1624] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:54:58,778 DEBUG [Thread-1624] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:54:58,871 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=413 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/C/bbcbcfd1299c4b9ea452dbbc0694a752 2023-04-19 20:54:58,876 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/A/6bb51a26606649e7b436816cc9f29277 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/6bb51a26606649e7b436816cc9f29277 2023-04-19 20:54:58,878 DEBUG [Thread-1624] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:54:58,879 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/6bb51a26606649e7b436816cc9f29277, entries=200, sequenceid=413, filesize=14.3 K 2023-04-19 20:54:58,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:54:58,880 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/B/e1801e9ceb804253841b6fdc70000bcc as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/e1801e9ceb804253841b6fdc70000bcc 2023-04-19 20:54:58,884 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/e1801e9ceb804253841b6fdc70000bcc, entries=100, sequenceid=413, filesize=9.5 K 2023-04-19 20:54:58,885 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/C/bbcbcfd1299c4b9ea452dbbc0694a752 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/bbcbcfd1299c4b9ea452dbbc0694a752 2023-04-19 20:54:58,892 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:54:58,895 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/bbcbcfd1299c4b9ea452dbbc0694a752, entries=100, sequenceid=413, filesize=9.5 K 2023-04-19 20:54:58,896 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 09bc544118dfd67701ac39d6da1a53f4 in 854ms, sequenceid=413, compaction requested=true 2023-04-19 20:54:58,896 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 09bc544118dfd67701ac39d6da1a53f4: 2023-04-19 20:54:58,896 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 20:54:58,896 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-19 20:54:58,896 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 20:54:58,897 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-19 20:54:58,897 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 20:54:58,897 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39540 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-19 20:54:58,897 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34660 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-19 20:54:58,898 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1912): 09bc544118dfd67701ac39d6da1a53f4/A is initiating minor compaction (all files) 2023-04-19 20:54:58,898 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1912): 09bc544118dfd67701ac39d6da1a53f4/B is initiating minor compaction (all files) 2023-04-19 20:54:58,898 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 09bc544118dfd67701ac39d6da1a53f4/A in TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. 2023-04-19 20:54:58,898 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 09bc544118dfd67701ac39d6da1a53f4/B in TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. 2023-04-19 20:54:58,898 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/fdd7f4955e3349349d002125a8a6eeec, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/dff48212e2c2468b9a91ef1ac8a770d0, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/6bb51a26606649e7b436816cc9f29277] into tmpdir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp, totalSize=38.6 K 2023-04-19 20:54:58,898 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/8fb851af21914f5793e7128f1612a95b, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/4893a404865c4f69aa7f8dbfb8bd3a08, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/e1801e9ceb804253841b6fdc70000bcc] into tmpdir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp, totalSize=33.8 K 2023-04-19 20:54:58,898 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting fdd7f4955e3349349d002125a8a6eeec, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=384, earliestPutTs=1681937694950 2023-04-19 20:54:58,898 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 8fb851af21914f5793e7128f1612a95b, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=384, earliestPutTs=1681937694950 2023-04-19 20:54:58,898 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting dff48212e2c2468b9a91ef1ac8a770d0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=400, earliestPutTs=1681937695905 2023-04-19 20:54:58,898 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 4893a404865c4f69aa7f8dbfb8bd3a08, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=400, earliestPutTs=1681937695905 2023-04-19 20:54:58,899 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 6bb51a26606649e7b436816cc9f29277, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=413, earliestPutTs=1681937698038 2023-04-19 20:54:58,899 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting e1801e9ceb804253841b6fdc70000bcc, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=413, earliestPutTs=1681937698041 2023-04-19 20:54:58,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:54:58,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:54:58,900 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:54:58,900 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:58,900 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:54:58,900 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:54:58,900 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:58,900 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:58,903 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:58,903 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:58,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:58,903 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:58,903 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:58,904 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:54:58,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:58,904 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:58,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:54:58,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:58,904 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:54:58,908 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:54:58,908 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:54:58,908 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:54:58,908 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:58,908 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:54:58,910 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:54:58,910 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:54:58,911 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. 2023-04-19 20:54:58,911 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. started... 2023-04-19 20:54:58,911 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 09bc544118dfd67701ac39d6da1a53f4 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-19 20:54:58,911 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] throttle.PressureAwareThroughputController(145): 09bc544118dfd67701ac39d6da1a53f4#A#compaction#255 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 57.69 MB/second 2023-04-19 20:54:58,911 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09bc544118dfd67701ac39d6da1a53f4, store=A 2023-04-19 20:54:58,911 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] throttle.PressureAwareThroughputController(145): 09bc544118dfd67701ac39d6da1a53f4#B#compaction#254 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 57.69 MB/second 2023-04-19 20:54:58,911 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 20:54:58,911 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09bc544118dfd67701ac39d6da1a53f4, store=B 2023-04-19 20:54:58,911 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 20:54:58,911 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09bc544118dfd67701ac39d6da1a53f4, store=C 2023-04-19 20:54:58,911 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 20:54:58,937 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=450 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/A/783ea13a8c224da8810e5073514b598d 2023-04-19 20:54:58,945 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=450 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/B/c13b49e4a2dd42109d7eeb8de2de4a6f 2023-04-19 20:54:59,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(9158): Flush requested on 09bc544118dfd67701ac39d6da1a53f4 2023-04-19 20:54:59,091 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. as already flushing 2023-04-19 20:54:59,110 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:59,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 282 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34548 deadline: 1681937759108, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:59,213 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:59,213 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:59,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34572 deadline: 1681937759211, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:59,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34540 deadline: 1681937759211, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:59,214 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:59,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 284 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34548 deadline: 1681937759211, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:59,326 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/A/b5ba1dc9236c477098de0284b97c2585 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/b5ba1dc9236c477098de0284b97c2585 2023-04-19 20:54:59,331 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 09bc544118dfd67701ac39d6da1a53f4/A of 09bc544118dfd67701ac39d6da1a53f4 into b5ba1dc9236c477098de0284b97c2585(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 20:54:59,331 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 09bc544118dfd67701ac39d6da1a53f4: 2023-04-19 20:54:59,331 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4., storeName=09bc544118dfd67701ac39d6da1a53f4/A, priority=13, startTime=1681937698896; duration=0sec 2023-04-19 20:54:59,331 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 20:54:59,331 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-19 20:54:59,332 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34626 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-19 20:54:59,332 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1912): 09bc544118dfd67701ac39d6da1a53f4/C is initiating minor compaction (all files) 2023-04-19 20:54:59,332 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 09bc544118dfd67701ac39d6da1a53f4/C in TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. 2023-04-19 20:54:59,332 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/082774a643da40d09836c6b732c57cfb, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/f1b48a2afa87427883ae753f87ddcfa6, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/bbcbcfd1299c4b9ea452dbbc0694a752] into tmpdir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp, totalSize=33.8 K 2023-04-19 20:54:59,333 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 082774a643da40d09836c6b732c57cfb, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=384, earliestPutTs=1681937694950 2023-04-19 20:54:59,333 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting f1b48a2afa87427883ae753f87ddcfa6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=400, earliestPutTs=1681937695905 2023-04-19 20:54:59,334 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting bbcbcfd1299c4b9ea452dbbc0694a752, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=413, earliestPutTs=1681937698041 2023-04-19 20:54:59,335 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/B/dd9479e8bd094a7ead99cbc4ab9c9b1a as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/dd9479e8bd094a7ead99cbc4ab9c9b1a 2023-04-19 20:54:59,340 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 09bc544118dfd67701ac39d6da1a53f4/B of 09bc544118dfd67701ac39d6da1a53f4 into dd9479e8bd094a7ead99cbc4ab9c9b1a(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 20:54:59,340 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 09bc544118dfd67701ac39d6da1a53f4: 2023-04-19 20:54:59,340 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4., storeName=09bc544118dfd67701ac39d6da1a53f4/B, priority=13, startTime=1681937698896; duration=0sec 2023-04-19 20:54:59,341 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 20:54:59,341 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] throttle.PressureAwareThroughputController(145): 09bc544118dfd67701ac39d6da1a53f4#C#compaction#259 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 57.69 MB/second 2023-04-19 20:54:59,350 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/C/cd25a11463aa4b3ab93befbd95930d81 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/cd25a11463aa4b3ab93befbd95930d81 2023-04-19 20:54:59,354 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 09bc544118dfd67701ac39d6da1a53f4/C of 09bc544118dfd67701ac39d6da1a53f4 into cd25a11463aa4b3ab93befbd95930d81(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 20:54:59,354 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 09bc544118dfd67701ac39d6da1a53f4: 2023-04-19 20:54:59,354 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4., storeName=09bc544118dfd67701ac39d6da1a53f4/C, priority=13, startTime=1681937698897; duration=0sec 2023-04-19 20:54:59,354 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 20:54:59,360 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=450 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/C/6816143ad73140c7bae602854de528b2 2023-04-19 20:54:59,370 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/A/783ea13a8c224da8810e5073514b598d as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/783ea13a8c224da8810e5073514b598d 2023-04-19 20:54:59,374 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/783ea13a8c224da8810e5073514b598d, entries=150, sequenceid=450, filesize=11.9 K 2023-04-19 20:54:59,375 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/B/c13b49e4a2dd42109d7eeb8de2de4a6f as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/c13b49e4a2dd42109d7eeb8de2de4a6f 2023-04-19 20:54:59,378 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/c13b49e4a2dd42109d7eeb8de2de4a6f, entries=150, sequenceid=450, filesize=11.9 K 2023-04-19 20:54:59,379 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/C/6816143ad73140c7bae602854de528b2 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/6816143ad73140c7bae602854de528b2 2023-04-19 20:54:59,383 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/6816143ad73140c7bae602854de528b2, entries=150, sequenceid=450, filesize=11.9 K 2023-04-19 20:54:59,384 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for 09bc544118dfd67701ac39d6da1a53f4 in 472ms, sequenceid=450, compaction requested=false 2023-04-19 20:54:59,384 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 09bc544118dfd67701ac39d6da1a53f4: 2023-04-19 20:54:59,384 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. 2023-04-19 20:54:59,384 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:54:59,384 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:54:59,384 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:59,384 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:54:59,384 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:54:59,385 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:59,385 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:59,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:59,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:59,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:59,386 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:59,386 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:54:59,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:59,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:59,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:59,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:59,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:59,387 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:54:59,387 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3eadb2e4[Count = 0] remaining members to acquire global barrier 2023-04-19 20:54:59,387 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:54:59,387 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:59,388 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:59,388 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:59,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:59,388 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:54:59,388 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:54:59,388 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:54:59,388 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:59,389 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:54:59,398 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:59,398 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:54:59,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:59,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:59,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:59,398 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:59,398 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:54:59,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:59,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:59,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:59,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:59,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:59,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:59,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:59,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:54:59,400 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:54:59,400 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:54:59,400 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:54:59,400 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:54:59,400 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:54:59,400 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:54:59,402 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:59,402 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:59,402 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:59,402 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:59,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:59,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:59,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:59,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:59,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:59,403 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:59,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:59,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:59,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:59,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:59,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:59,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:59,404 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:59,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:59,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:59,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:59,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:59,406 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:59,406 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:59,406 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:59,406 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:59,406 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:59,406 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:59,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:59,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:59,406 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:54:59,407 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:54:59,406 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:59,406 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:59,407 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:59,407 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:59,407 DEBUG [Thread-1624] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:54:59,407 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:54:59,407 DEBUG [Thread-1624] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:54:59,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:59,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:59,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(9158): Flush requested on 09bc544118dfd67701ac39d6da1a53f4 2023-04-19 20:54:59,416 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 09bc544118dfd67701ac39d6da1a53f4 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-19 20:54:59,416 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09bc544118dfd67701ac39d6da1a53f4, store=A 2023-04-19 20:54:59,416 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:54:59,416 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09bc544118dfd67701ac39d6da1a53f4, store=B 2023-04-19 20:54:59,416 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:54:59,416 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09bc544118dfd67701ac39d6da1a53f4, store=C 2023-04-19 20:54:59,416 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:54:59,461 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681937699445, max timestamp=1681937699459Num uniques 859; 2023-04-19 20:54:59,461 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681937699445, max timestamp=1681937699459Num uniques 859; 2023-04-19 20:54:59,461 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 20:54:59,461 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681937699445, max timestamp=1681937699459Num uniques 859; 2023-04-19 20:54:59,461 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 20:54:59,461 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 20:54:59,502 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:59,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 320 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34548 deadline: 1681937759500, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:59,507 DEBUG [Thread-1624] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:54:59,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:54:59,509 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:54:59,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:54:59,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:54:59,510 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:54:59,510 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:59,510 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:54:59,510 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:54:59,511 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:59,511 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:59,512 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:59,512 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:59,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:59,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:59,512 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:59,512 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:54:59,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:59,513 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:59,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:54:59,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:59,513 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:54:59,513 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:54:59,513 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:54:59,514 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:54:59,514 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:59,514 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:54:59,514 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:54:59,514 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:54:59,514 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. 2023-04-19 20:54:59,514 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. started... 2023-04-19 20:54:59,514 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. as already flushing 2023-04-19 20:54:59,514 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. 2023-04-19 20:54:59,514 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:54:59,514 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:54:59,514 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:59,514 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:54:59,514 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:54:59,518 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:59,518 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:59,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:59,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:59,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:59,518 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:59,518 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:54:59,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:59,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:59,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:59,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:59,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:59,519 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:54:59,519 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6f21eea0[Count = 0] remaining members to acquire global barrier 2023-04-19 20:54:59,520 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:54:59,520 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:59,521 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:59,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:59,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:59,521 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:54:59,521 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:54:59,521 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:54:59,521 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:59,521 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:54:59,522 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:59,522 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:54:59,522 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:59,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:59,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:59,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:59,522 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:54:59,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:59,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:59,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:59,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:59,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:59,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:59,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:59,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:54:59,524 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:54:59,524 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:54:59,525 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:54:59,525 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:54:59,525 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:54:59,525 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:54:59,526 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:59,526 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:59,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:59,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:59,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:59,526 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:59,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:59,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:59,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:59,527 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:59,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:59,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:59,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:59,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:59,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:59,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:59,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:59,528 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:59,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:59,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:59,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:59,532 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:59,532 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:59,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:59,532 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:59,532 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:59,532 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:54:59,532 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:54:59,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:59,532 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:59,532 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:59,532 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:59,532 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:59,532 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:59,532 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:59,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:59,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:59,532 DEBUG [Thread-1624] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:54:59,532 DEBUG [Thread-1624] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:54:59,604 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:59,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 322 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34548 deadline: 1681937759603, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:59,633 DEBUG [Thread-1624] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:54:59,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:54:59,635 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:54:59,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:54:59,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:54:59,636 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:54:59,636 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:59,636 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:54:59,636 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:54:59,636 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:59,636 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:59,637 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:59,637 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:59,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:59,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:59,638 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:59,638 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:54:59,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:59,638 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:59,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:54:59,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:59,638 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:54:59,639 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:54:59,639 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:54:59,639 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:54:59,639 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:59,639 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:54:59,639 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:54:59,639 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:54:59,639 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. 2023-04-19 20:54:59,639 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. started... 2023-04-19 20:54:59,639 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. as already flushing 2023-04-19 20:54:59,639 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. 2023-04-19 20:54:59,639 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:54:59,639 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:54:59,639 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:59,639 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:54:59,639 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:54:59,641 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:59,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:59,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:59,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:59,641 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:59,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:59,641 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:59,641 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:54:59,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:59,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:59,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:59,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:59,642 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:54:59,642 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1b1d4797[Count = 0] remaining members to acquire global barrier 2023-04-19 20:54:59,642 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:54:59,642 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:59,644 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:59,644 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:59,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:59,644 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:54:59,644 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:54:59,644 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:54:59,644 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:59,644 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:54:59,646 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:59,646 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:54:59,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:59,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:59,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:59,646 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:59,646 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:54:59,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:59,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:59,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:59,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:59,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:59,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:59,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:59,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:54:59,649 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:54:59,649 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:54:59,649 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:54:59,649 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:54:59,649 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:54:59,649 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:54:59,650 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:59,650 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:59,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:59,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:59,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:59,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:59,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:59,650 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:59,651 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:59,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:59,651 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:59,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:59,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:59,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:59,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:59,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:59,653 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:59,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:59,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:59,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:59,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:59,659 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:59,659 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:54:59,660 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:54:59,660 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:59,660 DEBUG [Thread-1624] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:54:59,660 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:59,660 DEBUG [Thread-1624] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:54:59,660 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:59,660 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:59,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:59,660 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:59,660 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:59,660 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:59,660 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:59,660 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:59,660 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:59,660 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:59,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:59,760 DEBUG [Thread-1624] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:54:59,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:54:59,762 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:54:59,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:54:59,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:54:59,763 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:54:59,763 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:59,763 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:54:59,763 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:54:59,764 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:59,764 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:59,765 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:59,765 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:59,765 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:59,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:59,765 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:59,765 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:54:59,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:59,766 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:59,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:54:59,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:59,766 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:54:59,766 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:54:59,767 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:54:59,767 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:54:59,767 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:59,767 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:54:59,767 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:54:59,767 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:54:59,767 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. 2023-04-19 20:54:59,767 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. started... 2023-04-19 20:54:59,767 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. as already flushing 2023-04-19 20:54:59,767 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. 2023-04-19 20:54:59,767 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:54:59,767 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:54:59,767 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:59,768 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:54:59,768 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:54:59,770 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:59,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:59,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:59,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:59,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:59,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:59,771 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:59,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:59,771 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:59,771 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:54:59,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:59,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:59,772 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:54:59,772 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@da51807[Count = 0] remaining members to acquire global barrier 2023-04-19 20:54:59,772 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:54:59,772 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:59,773 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:59,773 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:59,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:59,773 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:54:59,773 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:59,773 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:54:59,773 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:54:59,773 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:54:59,775 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:54:59,775 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:59,775 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:59,775 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:54:59,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:59,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:59,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:59,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:59,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:59,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:59,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:59,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:59,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:59,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:59,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:54:59,777 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:54:59,777 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:54:59,777 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:54:59,777 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:54:59,777 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:54:59,777 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:54:59,784 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:59,784 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:59,784 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:59,784 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:59,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:59,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:59,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:59,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:59,784 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:59,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:59,784 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:59,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:59,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:59,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:59,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:59,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:59,786 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:59,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:59,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:59,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:59,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:59,788 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:59,788 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:59,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:59,788 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:59,788 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:59,788 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:54:59,788 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:54:59,788 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:54:59,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:59,788 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:59,788 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:59,788 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:59,788 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:59,788 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:59,788 DEBUG [Thread-1624] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:54:59,789 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:59,789 DEBUG [Thread-1624] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:54:59,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:59,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:59,808 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:54:59,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 324 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34548 deadline: 1681937759805, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:59,825 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=464 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/A/3bbe895fafb545aea7c806d7715d344b 2023-04-19 20:54:59,834 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=464 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/B/bc12d65daff24a45aed8552f5eb61c2a 2023-04-19 20:54:59,889 DEBUG [Thread-1624] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:54:59,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:54:59,891 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:54:59,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:54:59,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:54:59,892 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:54:59,892 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:59,892 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:54:59,892 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:54:59,892 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:59,892 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:59,893 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:59,894 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:59,894 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:54:59,894 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:59,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:59,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:59,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:59,895 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:59,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:54:59,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:59,895 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:54:59,895 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:54:59,895 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:54:59,895 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:54:59,895 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:54:59,895 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:54:59,896 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:54:59,896 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:54:59,896 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. 2023-04-19 20:54:59,896 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. started... 2023-04-19 20:54:59,896 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. as already flushing 2023-04-19 20:54:59,896 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. 2023-04-19 20:54:59,896 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:54:59,896 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:54:59,896 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:59,896 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:54:59,896 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:54:59,897 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:59,897 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:59,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:59,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:59,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:59,898 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:59,898 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:54:59,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:59,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:59,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:59,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:59,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:59,899 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:54:59,899 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5e4ca1c5[Count = 0] remaining members to acquire global barrier 2023-04-19 20:54:59,899 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:54:59,899 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:59,901 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:59,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:59,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:59,901 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:54:59,901 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:54:59,901 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:54:59,901 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:59,901 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:54:59,903 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:59,903 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:54:59,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:59,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:59,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:59,903 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:59,903 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:54:59,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:59,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:59,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:59,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:59,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:59,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:59,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:59,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:54:59,905 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:54:59,905 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:54:59,905 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:54:59,905 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:54:59,905 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:54:59,905 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:54:59,906 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:59,906 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:59,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:59,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:54:59,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:54:59,906 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:59,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:59,906 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:59,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:54:59,907 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:59,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:59,907 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:59,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:59,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:59,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:54:59,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:59,909 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:59,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:59,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:54:59,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:54:59,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:59,911 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:54:59,911 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:59,911 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:54:59,911 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:54:59,911 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:59,911 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:54:59,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:54:59,911 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:54:59,911 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:54:59,911 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:54:59,911 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:54:59,911 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:54:59,911 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:59,911 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:54:59,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:54:59,911 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:54:59,911 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:54:59,912 DEBUG [Thread-1624] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:54:59,912 DEBUG [Thread-1624] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:00,012 DEBUG [Thread-1624] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:00,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:00,014 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:00,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:00,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:00,015 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:00,015 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:00,015 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:00,015 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:00,016 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:00,016 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:00,020 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:00,020 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:00,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:00,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:00,021 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:00,021 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:00,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:00,021 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:00,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:00,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:00,022 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:00,022 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:55:00,022 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:00,022 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:00,022 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:00,022 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:00,022 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:00,022 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:00,022 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. 2023-04-19 20:55:00,022 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. started... 2023-04-19 20:55:00,022 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. as already flushing 2023-04-19 20:55:00,023 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. 2023-04-19 20:55:00,023 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:00,023 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:00,023 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:00,023 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:00,023 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:00,024 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:00,024 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:00,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:00,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:00,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:00,025 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:00,025 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:00,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:00,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:00,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:00,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:00,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:00,026 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:00,026 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3ef81042[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:00,026 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:00,026 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:00,027 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:00,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:00,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:00,027 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:00,027 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:00,027 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:00,028 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:00,028 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:00,029 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:00,029 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:00,029 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:00,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:00,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:00,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:00,029 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:00,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:00,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:00,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:00,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:00,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:00,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:00,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:00,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:00,031 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:00,031 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:00,031 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:00,031 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:00,031 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:00,031 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:00,034 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:00,034 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:00,034 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:00,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:00,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:00,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:00,034 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:00,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:00,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:00,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:00,035 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:00,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:00,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:00,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:00,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:00,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:00,037 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:00,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:00,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:00,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:00,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:00,039 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:00,039 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:00,039 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:00,039 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:00,039 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:55:00,039 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:00,039 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:00,039 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:00,039 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:00,039 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:00,039 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:00,039 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:00,040 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:00,039 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:00,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:00,040 DEBUG [Thread-1624] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:00,040 DEBUG [Thread-1624] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:00,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:00,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:00,109 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:00,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 326 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34548 deadline: 1681937760109, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:00,140 DEBUG [Thread-1624] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:00,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:00,142 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:00,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:00,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:00,143 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:00,143 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:00,143 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:00,143 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:00,144 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:00,144 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:00,145 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:00,145 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:00,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:00,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:00,146 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:00,146 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:00,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:00,146 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:00,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:00,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:00,146 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:00,147 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:55:00,147 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:00,147 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:00,147 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:00,147 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:00,147 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:00,147 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:00,147 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. 2023-04-19 20:55:00,147 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. started... 2023-04-19 20:55:00,147 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. as already flushing 2023-04-19 20:55:00,147 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. 2023-04-19 20:55:00,148 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:00,148 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:00,148 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:00,148 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:00,148 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:00,149 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:00,149 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:00,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:00,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:00,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:00,150 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:00,150 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:00,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:00,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:00,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:00,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:00,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:00,151 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:00,151 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4fa8d9f5[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:00,151 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:00,151 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:00,154 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:00,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:00,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:00,154 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:00,154 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:00,154 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:00,154 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:00,154 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:00,156 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:00,156 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:00,156 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:00,156 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:00,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:00,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:00,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:00,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:00,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:00,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:00,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:00,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:00,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:00,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:00,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:00,158 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:00,158 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:00,158 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:00,158 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:00,158 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:00,158 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:00,160 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:00,160 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:00,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:00,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:00,160 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:00,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:00,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:00,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:00,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:00,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:00,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:00,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:00,161 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:00,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:00,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:00,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:00,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:00,163 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:00,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:00,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:00,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:00,168 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:00,168 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:00,168 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:00,168 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:00,168 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:00,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:00,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:00,168 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:00,168 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:00,168 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:00,168 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:00,168 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:00,168 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:00,168 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:00,168 DEBUG [Thread-1624] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:00,168 DEBUG [Thread-1624] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:00,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:00,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:00,219 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:00,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34540 deadline: 1681937760217, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:00,220 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:00,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34572 deadline: 1681937760218, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:00,249 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=464 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/C/4e2ede360c804aaca211219533c9a658 2023-04-19 20:55:00,253 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/A/3bbe895fafb545aea7c806d7715d344b as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/3bbe895fafb545aea7c806d7715d344b 2023-04-19 20:55:00,258 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/3bbe895fafb545aea7c806d7715d344b, entries=150, sequenceid=464, filesize=11.9 K 2023-04-19 20:55:00,259 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/B/bc12d65daff24a45aed8552f5eb61c2a as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/bc12d65daff24a45aed8552f5eb61c2a 2023-04-19 20:55:00,262 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/bc12d65daff24a45aed8552f5eb61c2a, entries=100, sequenceid=464, filesize=9.5 K 2023-04-19 20:55:00,263 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/C/4e2ede360c804aaca211219533c9a658 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/4e2ede360c804aaca211219533c9a658 2023-04-19 20:55:00,266 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/4e2ede360c804aaca211219533c9a658, entries=100, sequenceid=464, filesize=9.5 K 2023-04-19 20:55:00,267 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 09bc544118dfd67701ac39d6da1a53f4 in 851ms, sequenceid=464, compaction requested=true 2023-04-19 20:55:00,267 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 09bc544118dfd67701ac39d6da1a53f4: 2023-04-19 20:55:00,267 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 20:55:00,267 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 20:55:00,267 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-19 20:55:00,267 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-19 20:55:00,268 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-19 20:55:00,268 DEBUG [Thread-1624] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:00,269 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37202 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-19 20:55:00,269 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1912): 09bc544118dfd67701ac39d6da1a53f4/A is initiating minor compaction (all files) 2023-04-19 20:55:00,269 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 09bc544118dfd67701ac39d6da1a53f4/A in TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. 2023-04-19 20:55:00,269 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/b5ba1dc9236c477098de0284b97c2585, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/783ea13a8c224da8810e5073514b598d, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/3bbe895fafb545aea7c806d7715d344b] into tmpdir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp, totalSize=36.3 K 2023-04-19 20:55:00,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:00,269 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34762 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-19 20:55:00,269 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1912): 09bc544118dfd67701ac39d6da1a53f4/B is initiating minor compaction (all files) 2023-04-19 20:55:00,270 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 09bc544118dfd67701ac39d6da1a53f4/B in TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. 2023-04-19 20:55:00,270 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/dd9479e8bd094a7ead99cbc4ab9c9b1a, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/c13b49e4a2dd42109d7eeb8de2de4a6f, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/bc12d65daff24a45aed8552f5eb61c2a] into tmpdir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp, totalSize=33.9 K 2023-04-19 20:55:00,270 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting b5ba1dc9236c477098de0284b97c2585, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=413, earliestPutTs=1681937695909 2023-04-19 20:55:00,270 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting dd9479e8bd094a7ead99cbc4ab9c9b1a, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=413, earliestPutTs=1681937695909 2023-04-19 20:55:00,271 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 783ea13a8c224da8810e5073514b598d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=450, earliestPutTs=1681937698088 2023-04-19 20:55:00,271 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting c13b49e4a2dd42109d7eeb8de2de4a6f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=450, earliestPutTs=1681937698088 2023-04-19 20:55:00,271 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 3bbe895fafb545aea7c806d7715d344b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=464, earliestPutTs=1681937699096 2023-04-19 20:55:00,271 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting bc12d65daff24a45aed8552f5eb61c2a, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=464, earliestPutTs=1681937699107 2023-04-19 20:55:00,272 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:00,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:00,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:00,277 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:00,277 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:00,278 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:00,278 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:00,278 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:00,278 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:00,279 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:00,279 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:00,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:00,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:00,280 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:00,280 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:00,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:00,280 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] throttle.PressureAwareThroughputController(145): 09bc544118dfd67701ac39d6da1a53f4#A#compaction#263 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 57.69 MB/second 2023-04-19 20:55:00,280 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:00,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:00,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:00,281 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:00,281 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:55:00,281 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:00,281 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:00,281 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:00,281 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:00,281 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:00,282 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:00,282 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. 2023-04-19 20:55:00,282 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. started... 2023-04-19 20:55:00,282 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 09bc544118dfd67701ac39d6da1a53f4 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-04-19 20:55:00,282 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09bc544118dfd67701ac39d6da1a53f4, store=A 2023-04-19 20:55:00,282 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 20:55:00,282 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09bc544118dfd67701ac39d6da1a53f4, store=B 2023-04-19 20:55:00,282 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 20:55:00,282 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09bc544118dfd67701ac39d6da1a53f4, store=C 2023-04-19 20:55:00,282 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 20:55:00,282 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] throttle.PressureAwareThroughputController(145): 09bc544118dfd67701ac39d6da1a53f4#B#compaction#264 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 57.69 MB/second 2023-04-19 20:55:00,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(9158): Flush requested on 09bc544118dfd67701ac39d6da1a53f4 2023-04-19 20:55:00,612 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. as already flushing 2023-04-19 20:55:00,637 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:00,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 336 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34548 deadline: 1681937760635, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:00,694 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/A/369cec6160634e64a3883168ae3ee266 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/369cec6160634e64a3883168ae3ee266 2023-04-19 20:55:00,702 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=500 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/A/9990a212308145d1a151976c97e4c009 2023-04-19 20:55:00,707 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 09bc544118dfd67701ac39d6da1a53f4/A of 09bc544118dfd67701ac39d6da1a53f4 into 369cec6160634e64a3883168ae3ee266(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 20:55:00,707 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 09bc544118dfd67701ac39d6da1a53f4: 2023-04-19 20:55:00,707 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4., storeName=09bc544118dfd67701ac39d6da1a53f4/A, priority=13, startTime=1681937700267; duration=0sec 2023-04-19 20:55:00,708 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 20:55:00,708 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-19 20:55:00,709 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34728 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-19 20:55:00,709 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1912): 09bc544118dfd67701ac39d6da1a53f4/C is initiating minor compaction (all files) 2023-04-19 20:55:00,709 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 09bc544118dfd67701ac39d6da1a53f4/C in TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. 2023-04-19 20:55:00,709 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/cd25a11463aa4b3ab93befbd95930d81, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/6816143ad73140c7bae602854de528b2, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/4e2ede360c804aaca211219533c9a658] into tmpdir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp, totalSize=33.9 K 2023-04-19 20:55:00,710 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting cd25a11463aa4b3ab93befbd95930d81, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=413, earliestPutTs=1681937695909 2023-04-19 20:55:00,710 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/B/154ffbdf7a8c492ca2d85d67a5172496 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/154ffbdf7a8c492ca2d85d67a5172496 2023-04-19 20:55:00,710 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 6816143ad73140c7bae602854de528b2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=450, earliestPutTs=1681937698088 2023-04-19 20:55:00,711 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 4e2ede360c804aaca211219533c9a658, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=464, earliestPutTs=1681937699107 2023-04-19 20:55:00,717 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 09bc544118dfd67701ac39d6da1a53f4/B of 09bc544118dfd67701ac39d6da1a53f4 into 154ffbdf7a8c492ca2d85d67a5172496(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 20:55:00,717 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 09bc544118dfd67701ac39d6da1a53f4: 2023-04-19 20:55:00,717 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4., storeName=09bc544118dfd67701ac39d6da1a53f4/B, priority=13, startTime=1681937700267; duration=0sec 2023-04-19 20:55:00,718 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 20:55:00,722 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] throttle.PressureAwareThroughputController(145): 09bc544118dfd67701ac39d6da1a53f4#C#compaction#267 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 57.69 MB/second 2023-04-19 20:55:00,740 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:00,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 338 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34548 deadline: 1681937760739, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:00,943 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:00,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 340 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34548 deadline: 1681937760941, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:01,146 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=500 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/B/481b8141ee7a4f2fa36950f0326bc099 2023-04-19 20:55:01,151 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/C/08ccad336b694a10be95d9e4f06d5367 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/08ccad336b694a10be95d9e4f06d5367 2023-04-19 20:55:01,160 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 09bc544118dfd67701ac39d6da1a53f4/C of 09bc544118dfd67701ac39d6da1a53f4 into 08ccad336b694a10be95d9e4f06d5367(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 20:55:01,160 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 09bc544118dfd67701ac39d6da1a53f4: 2023-04-19 20:55:01,160 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4., storeName=09bc544118dfd67701ac39d6da1a53f4/C, priority=13, startTime=1681937700267; duration=0sec 2023-04-19 20:55:01,161 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 20:55:01,247 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:01,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 342 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34548 deadline: 1681937761245, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:01,560 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=500 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/C/8fc10a4ad5d74787a8db2ef8422504f3 2023-04-19 20:55:01,565 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/A/9990a212308145d1a151976c97e4c009 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/9990a212308145d1a151976c97e4c009 2023-04-19 20:55:01,568 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/9990a212308145d1a151976c97e4c009, entries=150, sequenceid=500, filesize=11.9 K 2023-04-19 20:55:01,569 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/B/481b8141ee7a4f2fa36950f0326bc099 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/481b8141ee7a4f2fa36950f0326bc099 2023-04-19 20:55:01,572 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/481b8141ee7a4f2fa36950f0326bc099, entries=150, sequenceid=500, filesize=11.9 K 2023-04-19 20:55:01,573 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/C/8fc10a4ad5d74787a8db2ef8422504f3 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/8fc10a4ad5d74787a8db2ef8422504f3 2023-04-19 20:55:01,577 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/8fc10a4ad5d74787a8db2ef8422504f3, entries=150, sequenceid=500, filesize=11.9 K 2023-04-19 20:55:01,577 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=53.67 KB/54960 for 09bc544118dfd67701ac39d6da1a53f4 in 1295ms, sequenceid=500, compaction requested=false 2023-04-19 20:55:01,577 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 09bc544118dfd67701ac39d6da1a53f4: 2023-04-19 20:55:01,577 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. 2023-04-19 20:55:01,577 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:01,578 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:01,578 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:01,578 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:01,578 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:01,579 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:01,579 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:01,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:01,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:01,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:01,579 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:01,580 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:01,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:01,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:01,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:01,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:01,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:01,581 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:01,581 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a688527[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:01,582 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:01,582 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:01,583 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:01,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:01,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:01,583 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:01,583 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:01,583 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:01,583 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:01,583 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:01,585 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:01,585 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:01,585 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:01,585 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:01,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:01,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:01,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:01,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:01,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:01,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:01,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:01,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:01,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:01,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:01,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:01,587 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:01,587 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:01,587 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:01,587 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:01,587 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:01,587 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:01,589 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:01,589 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:01,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:01,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:01,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:01,589 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:01,589 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:01,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:01,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:01,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:01,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:01,590 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:01,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:01,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:01,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:01,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:01,591 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:01,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:01,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:01,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:01,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:01,594 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:01,594 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:01,594 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:01,594 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:01,594 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:01,594 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:01,594 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:01,594 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:01,594 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:01,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:01,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:01,594 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:01,594 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:01,594 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:01,594 DEBUG [Thread-1624] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:01,594 DEBUG [Thread-1624] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:01,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:01,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:01,695 DEBUG [Thread-1624] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:01,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:01,697 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:01,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:01,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:01,698 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:01,698 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:01,698 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:01,698 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:01,698 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:01,698 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:01,700 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:01,700 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:01,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:01,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:01,700 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:01,701 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:01,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:01,701 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:01,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:01,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:01,701 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:01,701 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:55:01,701 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:01,701 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:01,702 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:01,702 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:01,702 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:01,702 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:01,702 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. 2023-04-19 20:55:01,702 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. started... 2023-04-19 20:55:01,702 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 09bc544118dfd67701ac39d6da1a53f4 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-19 20:55:01,702 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09bc544118dfd67701ac39d6da1a53f4, store=A 2023-04-19 20:55:01,702 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:55:01,702 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09bc544118dfd67701ac39d6da1a53f4, store=B 2023-04-19 20:55:01,702 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:55:01,702 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09bc544118dfd67701ac39d6da1a53f4, store=C 2023-04-19 20:55:01,702 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:55:01,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(9158): Flush requested on 09bc544118dfd67701ac39d6da1a53f4 2023-04-19 20:55:01,749 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. as already flushing 2023-04-19 20:55:01,804 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681937701775, max timestamp=1681937701802Num uniques 859; 2023-04-19 20:55:01,804 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 20:55:01,804 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681937701775, max timestamp=1681937701802Num uniques 859; 2023-04-19 20:55:01,804 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 20:55:01,805 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681937701775, max timestamp=1681937701802Num uniques 859; 2023-04-19 20:55:01,805 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 20:55:01,854 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:01,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 377 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34548 deadline: 1681937761852, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:01,957 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:01,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 379 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34548 deadline: 1681937761955, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:02,111 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=514 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/A/6fc4c2b04bbf4f3aabbe68e6e85d9bfd 2023-04-19 20:55:02,120 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=514 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/B/2813c1781ec84f0c8894113d55713faf 2023-04-19 20:55:02,161 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:02,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 381 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34548 deadline: 1681937762159, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:02,235 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:02,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34540 deadline: 1681937762233, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:02,235 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:02,236 DEBUG [Thread-1618] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4140 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4., hostname=jenkins-hbase4.apache.org,35247,1681937616971, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-19 20:55:02,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34572 deadline: 1681937762234, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:02,238 DEBUG [Thread-1614] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4143 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4., hostname=jenkins-hbase4.apache.org,35247,1681937616971, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-19 20:55:02,463 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:02,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 383 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34548 deadline: 1681937762463, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:02,553 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=514 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/C/7d6045f912414592bad9403bb9de6176 2023-04-19 20:55:02,557 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/A/6fc4c2b04bbf4f3aabbe68e6e85d9bfd as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/6fc4c2b04bbf4f3aabbe68e6e85d9bfd 2023-04-19 20:55:02,560 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/6fc4c2b04bbf4f3aabbe68e6e85d9bfd, entries=150, sequenceid=514, filesize=11.9 K 2023-04-19 20:55:02,561 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/B/2813c1781ec84f0c8894113d55713faf as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/2813c1781ec84f0c8894113d55713faf 2023-04-19 20:55:02,565 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/2813c1781ec84f0c8894113d55713faf, entries=150, sequenceid=514, filesize=11.9 K 2023-04-19 20:55:02,566 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/C/7d6045f912414592bad9403bb9de6176 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/7d6045f912414592bad9403bb9de6176 2023-04-19 20:55:02,569 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/7d6045f912414592bad9403bb9de6176, entries=150, sequenceid=514, filesize=11.9 K 2023-04-19 20:55:02,570 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 09bc544118dfd67701ac39d6da1a53f4 in 868ms, sequenceid=514, compaction requested=true 2023-04-19 20:55:02,570 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 09bc544118dfd67701ac39d6da1a53f4: 2023-04-19 20:55:02,570 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. 2023-04-19 20:55:02,570 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:02,570 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:02,570 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:02,570 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:02,570 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:02,572 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:02,572 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:02,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:02,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:02,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:02,573 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:02,573 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:02,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:02,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:02,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:02,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:02,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:02,574 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:02,574 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@51114326[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:02,574 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:02,574 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:02,576 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:02,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:02,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:02,576 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:02,576 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:02,576 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:02,576 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:02,576 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:02,577 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:02,577 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:02,577 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:02,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:02,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:02,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:02,577 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:02,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:02,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:02,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:02,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:02,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:02,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:02,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:02,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:02,580 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:02,580 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:02,580 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:02,580 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:02,580 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:02,580 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:02,581 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:02,581 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:02,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:02,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:02,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:02,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:02,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:02,582 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:02,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:02,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:02,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:02,582 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:02,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:02,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:02,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:02,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:02,583 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:02,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:02,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:02,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:02,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:02,586 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:02,586 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:02,586 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:02,586 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:02,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:02,586 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:02,586 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:55:02,586 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:02,586 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:02,586 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:02,587 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:02,587 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:02,587 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:02,587 DEBUG [Thread-1624] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:02,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:02,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:02,587 DEBUG [Thread-1624] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:02,587 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:02,587 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:02,687 DEBUG [Thread-1624] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:02,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:02,689 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:02,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:02,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:02,690 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:02,690 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:02,690 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:02,690 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:02,691 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:02,691 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:02,692 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:02,692 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:02,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:02,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:02,693 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:02,693 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:02,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:02,693 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:02,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:02,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:02,694 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:02,694 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:55:02,694 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:02,694 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:02,694 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:02,694 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:02,694 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:02,694 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:02,694 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. 2023-04-19 20:55:02,694 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. started... 2023-04-19 20:55:02,694 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 09bc544118dfd67701ac39d6da1a53f4 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-04-19 20:55:02,695 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09bc544118dfd67701ac39d6da1a53f4, store=A 2023-04-19 20:55:02,695 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 20:55:02,695 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09bc544118dfd67701ac39d6da1a53f4, store=B 2023-04-19 20:55:02,695 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 20:55:02,695 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09bc544118dfd67701ac39d6da1a53f4, store=C 2023-04-19 20:55:02,695 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 20:55:02,703 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=550 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/A/c41b3bdc48f6425b837de01a022fcf01 2023-04-19 20:55:02,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(9158): Flush requested on 09bc544118dfd67701ac39d6da1a53f4 2023-04-19 20:55:02,971 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. as already flushing 2023-04-19 20:55:02,997 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:02,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 393 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34548 deadline: 1681937762994, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:03,099 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:03,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 395 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34548 deadline: 1681937763098, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:03,114 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=550 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/B/a79caa3698db471280ee302caf05e4ee 2023-04-19 20:55:03,126 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=550 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/C/29905e6bdd3e4ce2b9fe5182aaa67b9b 2023-04-19 20:55:03,129 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/A/c41b3bdc48f6425b837de01a022fcf01 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/c41b3bdc48f6425b837de01a022fcf01 2023-04-19 20:55:03,133 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/c41b3bdc48f6425b837de01a022fcf01, entries=150, sequenceid=550, filesize=11.9 K 2023-04-19 20:55:03,134 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/B/a79caa3698db471280ee302caf05e4ee as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/a79caa3698db471280ee302caf05e4ee 2023-04-19 20:55:03,137 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/a79caa3698db471280ee302caf05e4ee, entries=150, sequenceid=550, filesize=11.9 K 2023-04-19 20:55:03,138 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/C/29905e6bdd3e4ce2b9fe5182aaa67b9b as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/29905e6bdd3e4ce2b9fe5182aaa67b9b 2023-04-19 20:55:03,141 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/29905e6bdd3e4ce2b9fe5182aaa67b9b, entries=150, sequenceid=550, filesize=11.9 K 2023-04-19 20:55:03,142 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=53.67 KB/54960 for 09bc544118dfd67701ac39d6da1a53f4 in 448ms, sequenceid=550, compaction requested=true 2023-04-19 20:55:03,142 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 09bc544118dfd67701ac39d6da1a53f4: 2023-04-19 20:55:03,142 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. 2023-04-19 20:55:03,142 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:03,142 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:03,142 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:03,142 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:03,142 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:03,143 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:03,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:03,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:03,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:03,143 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:03,144 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:03,144 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:03,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:03,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:03,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:03,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:03,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:03,145 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:03,145 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2f153346[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:03,145 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:03,145 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:03,146 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:03,146 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:03,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:03,146 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:03,146 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:03,146 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:03,146 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:03,147 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:03,147 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:03,147 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:03,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:03,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:03,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:03,148 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:03,148 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:03,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:03,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:03,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:03,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:03,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:03,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:03,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:03,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:03,150 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:03,150 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:03,150 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:03,150 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:03,150 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:03,150 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:03,153 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:03,153 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:03,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:03,153 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:03,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:03,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:03,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:03,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:03,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:03,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:03,153 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:03,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:03,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:03,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:03,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:03,154 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:03,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:03,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:03,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:03,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:03,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:03,156 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:03,156 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:03,156 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:03,156 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:03,156 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:03,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:03,156 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:03,157 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:03,156 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:03,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:03,156 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:03,157 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:55:03,157 DEBUG [Thread-1624] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:03,156 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:03,157 DEBUG [Thread-1624] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:03,157 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:03,157 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:03,157 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:03,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:03,257 DEBUG [Thread-1624] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:03,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:03,259 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:03,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:03,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:03,260 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:03,260 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:03,260 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:03,260 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:03,261 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:03,261 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:03,265 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:03,265 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:03,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:03,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:03,265 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:03,265 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:03,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:03,266 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:03,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:03,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:03,266 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:03,266 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:55:03,266 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:03,266 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:03,266 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:03,266 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:03,266 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:03,266 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:03,266 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. 2023-04-19 20:55:03,267 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. started... 2023-04-19 20:55:03,267 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 09bc544118dfd67701ac39d6da1a53f4 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-19 20:55:03,267 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09bc544118dfd67701ac39d6da1a53f4, store=A 2023-04-19 20:55:03,267 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:55:03,267 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09bc544118dfd67701ac39d6da1a53f4, store=B 2023-04-19 20:55:03,267 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:55:03,267 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09bc544118dfd67701ac39d6da1a53f4, store=C 2023-04-19 20:55:03,267 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:55:03,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(9158): Flush requested on 09bc544118dfd67701ac39d6da1a53f4 2023-04-19 20:55:03,302 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. as already flushing 2023-04-19 20:55:03,360 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681937703345, max timestamp=1681937703358Num uniques 859; 2023-04-19 20:55:03,360 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 20:55:03,360 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681937703345, max timestamp=1681937703358Num uniques 859; 2023-04-19 20:55:03,360 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681937703345, max timestamp=1681937703358Num uniques 859; 2023-04-19 20:55:03,360 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 20:55:03,360 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 20:55:03,416 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:03,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 430 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34548 deadline: 1681937763411, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:03,519 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:03,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 432 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34548 deadline: 1681937763517, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:03,684 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=561 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/A/df3865b5cdd74e4b920d3039de3ba6d5 2023-04-19 20:55:03,696 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=561 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/B/6475b28a544a423c91215ef654acdc1c 2023-04-19 20:55:03,721 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:03,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 434 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34548 deadline: 1681937763720, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:04,025 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:04,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 436 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34548 deadline: 1681937764024, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:04,107 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=561 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/C/75086013405f483c86c594069f6275f6 2023-04-19 20:55:04,111 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/A/df3865b5cdd74e4b920d3039de3ba6d5 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/df3865b5cdd74e4b920d3039de3ba6d5 2023-04-19 20:55:04,114 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/df3865b5cdd74e4b920d3039de3ba6d5, entries=150, sequenceid=561, filesize=11.9 K 2023-04-19 20:55:04,115 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/B/6475b28a544a423c91215ef654acdc1c as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/6475b28a544a423c91215ef654acdc1c 2023-04-19 20:55:04,118 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/6475b28a544a423c91215ef654acdc1c, entries=150, sequenceid=561, filesize=11.9 K 2023-04-19 20:55:04,119 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/C/75086013405f483c86c594069f6275f6 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/75086013405f483c86c594069f6275f6 2023-04-19 20:55:04,122 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/75086013405f483c86c594069f6275f6, entries=150, sequenceid=561, filesize=11.9 K 2023-04-19 20:55:04,123 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 09bc544118dfd67701ac39d6da1a53f4 in 856ms, sequenceid=561, compaction requested=true 2023-04-19 20:55:04,123 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 09bc544118dfd67701ac39d6da1a53f4: 2023-04-19 20:55:04,123 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. 2023-04-19 20:55:04,123 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:04,123 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:04,123 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:04,124 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:04,124 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:04,126 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:04,126 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:04,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:04,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:04,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:04,126 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:04,126 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:04,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:04,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:04,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:04,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:04,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:04,128 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:04,128 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@26fb2020[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:04,128 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:04,128 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:04,129 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:04,129 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:04,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:04,129 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:04,129 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:04,129 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:04,129 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:04,129 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:04,130 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:04,130 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:04,130 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:04,130 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:04,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:04,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:04,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:04,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:04,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:04,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:04,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:04,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:04,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:04,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:04,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:04,133 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:04,133 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:04,133 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:04,133 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:04,133 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:04,133 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:04,135 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:04,135 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:04,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:04,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:04,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:04,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:04,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:04,135 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:04,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:04,135 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:04,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:04,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:04,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:04,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:04,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:04,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:04,136 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:04,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:04,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:04,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:04,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:04,140 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:04,140 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:04,140 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:04,140 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:04,140 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:04,140 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:04,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:04,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:04,140 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:04,140 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:04,140 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:04,140 DEBUG [Thread-1624] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:04,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:04,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:04,140 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:04,141 DEBUG [Thread-1624] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:04,141 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:04,141 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:04,241 DEBUG [Thread-1624] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:04,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:04,243 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:04,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:04,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:04,244 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:04,244 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:04,244 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:04,244 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:04,244 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:04,244 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:04,246 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:04,246 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:04,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:04,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:04,246 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:04,246 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:04,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:04,246 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:04,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:04,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:04,247 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:04,247 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:55:04,247 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:04,247 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:04,247 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:04,247 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:04,247 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:04,247 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:04,247 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. 2023-04-19 20:55:04,247 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. started... 2023-04-19 20:55:04,247 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 09bc544118dfd67701ac39d6da1a53f4 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-04-19 20:55:04,248 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09bc544118dfd67701ac39d6da1a53f4, store=A 2023-04-19 20:55:04,248 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 20:55:04,248 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09bc544118dfd67701ac39d6da1a53f4, store=B 2023-04-19 20:55:04,248 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 20:55:04,248 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09bc544118dfd67701ac39d6da1a53f4, store=C 2023-04-19 20:55:04,248 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 20:55:04,255 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=597 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/A/c1be002333a7466388824e4d77b119e1 2023-04-19 20:55:04,264 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=597 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/B/581204aa24eb48b5bc935992d8083a2a 2023-04-19 20:55:04,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(9158): Flush requested on 09bc544118dfd67701ac39d6da1a53f4 2023-04-19 20:55:04,532 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. as already flushing 2023-04-19 20:55:04,564 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:04,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 446 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34548 deadline: 1681937764561, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:04,667 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:04,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 448 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34548 deadline: 1681937764665, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:04,673 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=597 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/C/9850070a77314694a6257864c0c092bf 2023-04-19 20:55:04,677 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/A/c1be002333a7466388824e4d77b119e1 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/c1be002333a7466388824e4d77b119e1 2023-04-19 20:55:04,680 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/c1be002333a7466388824e4d77b119e1, entries=150, sequenceid=597, filesize=11.9 K 2023-04-19 20:55:04,681 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/B/581204aa24eb48b5bc935992d8083a2a as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/581204aa24eb48b5bc935992d8083a2a 2023-04-19 20:55:04,685 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/581204aa24eb48b5bc935992d8083a2a, entries=150, sequenceid=597, filesize=11.9 K 2023-04-19 20:55:04,685 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/C/9850070a77314694a6257864c0c092bf as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/9850070a77314694a6257864c0c092bf 2023-04-19 20:55:04,689 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/9850070a77314694a6257864c0c092bf, entries=150, sequenceid=597, filesize=11.9 K 2023-04-19 20:55:04,689 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=53.67 KB/54960 for 09bc544118dfd67701ac39d6da1a53f4 in 442ms, sequenceid=597, compaction requested=true 2023-04-19 20:55:04,689 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 09bc544118dfd67701ac39d6da1a53f4: 2023-04-19 20:55:04,689 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. 2023-04-19 20:55:04,690 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:04,690 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:04,690 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:04,690 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:04,690 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:04,691 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:04,691 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:04,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:04,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:04,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:04,694 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:04,694 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:04,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:04,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:04,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:04,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:04,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:04,696 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:04,696 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@894c865[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:04,696 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:04,696 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:04,697 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:04,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:04,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:04,697 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:04,697 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:04,697 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:04,697 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:04,697 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:04,699 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:04,699 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:04,699 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:04,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:04,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:04,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:04,699 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:04,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:04,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:04,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:04,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:04,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:04,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:04,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:04,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:04,702 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:04,702 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:04,702 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:04,702 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:04,702 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:04,702 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:04,703 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:04,703 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:04,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:04,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:04,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:04,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:04,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:04,703 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:04,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:04,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:04,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:04,704 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:04,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:04,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:04,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:04,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:04,705 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:04,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:04,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:04,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:04,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:04,707 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:04,707 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:04,707 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:04,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:04,707 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:04,707 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:04,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:04,707 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:04,707 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:04,707 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:04,707 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:04,707 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:55:04,707 DEBUG [Thread-1624] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:04,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:04,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:04,707 DEBUG [Thread-1624] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:04,707 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:04,707 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:04,707 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:04,807 DEBUG [Thread-1624] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:04,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:04,809 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:04,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:04,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:04,811 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:04,811 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:04,811 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:04,811 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:04,811 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:04,811 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:04,813 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:04,813 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:04,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:04,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:04,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:04,813 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:04,813 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:04,813 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:04,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:04,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:04,814 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:04,814 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:55:04,814 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:04,814 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:04,814 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:04,814 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:04,814 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:04,815 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:04,815 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. 2023-04-19 20:55:04,815 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. started... 2023-04-19 20:55:04,815 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 09bc544118dfd67701ac39d6da1a53f4 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-19 20:55:04,815 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09bc544118dfd67701ac39d6da1a53f4, store=A 2023-04-19 20:55:04,815 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:55:04,815 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09bc544118dfd67701ac39d6da1a53f4, store=B 2023-04-19 20:55:04,815 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:55:04,815 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09bc544118dfd67701ac39d6da1a53f4, store=C 2023-04-19 20:55:04,815 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:55:04,824 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=608 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/A/20d8c2d9c1af44bca0bdde146ba7402d 2023-04-19 20:55:04,835 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=608 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/B/2841a6cdafab4f7f9a8569decde21f5b 2023-04-19 20:55:04,844 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=608 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/C/889f355d12ed4a2b8fc58a102c8d1d91 2023-04-19 20:55:04,848 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/A/20d8c2d9c1af44bca0bdde146ba7402d as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/20d8c2d9c1af44bca0bdde146ba7402d 2023-04-19 20:55:04,852 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/20d8c2d9c1af44bca0bdde146ba7402d, entries=150, sequenceid=608, filesize=11.9 K 2023-04-19 20:55:04,852 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/B/2841a6cdafab4f7f9a8569decde21f5b as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/2841a6cdafab4f7f9a8569decde21f5b 2023-04-19 20:55:04,856 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/2841a6cdafab4f7f9a8569decde21f5b, entries=150, sequenceid=608, filesize=11.9 K 2023-04-19 20:55:04,857 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/C/889f355d12ed4a2b8fc58a102c8d1d91 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/889f355d12ed4a2b8fc58a102c8d1d91 2023-04-19 20:55:04,860 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/889f355d12ed4a2b8fc58a102c8d1d91, entries=150, sequenceid=608, filesize=11.9 K 2023-04-19 20:55:04,860 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=0 B/0 for 09bc544118dfd67701ac39d6da1a53f4 in 45ms, sequenceid=608, compaction requested=true 2023-04-19 20:55:04,861 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 09bc544118dfd67701ac39d6da1a53f4: 2023-04-19 20:55:04,861 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. 2023-04-19 20:55:04,861 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:04,861 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:04,861 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:04,861 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:04,861 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:04,863 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:04,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:04,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:04,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:04,864 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:04,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:04,864 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:04,864 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:04,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:04,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:04,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:04,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:04,865 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:04,865 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5347a222[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:04,865 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:04,865 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:04,867 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:04,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:04,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:04,867 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:04,867 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:04,867 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:04,867 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:04,867 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:04,868 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:04,868 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:04,868 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:04,868 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:04,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:04,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:04,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:04,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:04,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:04,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:04,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:04,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:04,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:04,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:04,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:04,871 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:04,871 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:04,871 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:04,871 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:04,871 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:04,871 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:04,873 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:04,873 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:04,873 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:04,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:04,873 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:04,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:04,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:04,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:04,873 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:04,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:04,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:04,874 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:04,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:04,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:04,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:04,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:04,875 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:04,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:04,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:04,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:04,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:04,878 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:04,878 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:04,878 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:04,878 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:04,878 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:04,878 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:04,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:04,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:04,878 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:04,878 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:04,878 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:04,878 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:04,878 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:04,878 DEBUG [Thread-1624] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:04,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:04,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:04,878 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:04,878 DEBUG [Thread-1624] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:04,878 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:55:04,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(9158): Flush requested on 09bc544118dfd67701ac39d6da1a53f4 2023-04-19 20:55:04,890 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 09bc544118dfd67701ac39d6da1a53f4 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-19 20:55:04,890 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09bc544118dfd67701ac39d6da1a53f4, store=A 2023-04-19 20:55:04,890 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:55:04,890 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09bc544118dfd67701ac39d6da1a53f4, store=B 2023-04-19 20:55:04,891 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:55:04,891 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09bc544118dfd67701ac39d6da1a53f4, store=C 2023-04-19 20:55:04,891 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:55:04,963 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681937704945, max timestamp=1681937704962Num uniques 859; 2023-04-19 20:55:04,963 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 20:55:04,964 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681937704945, max timestamp=1681937704962Num uniques 859; 2023-04-19 20:55:04,964 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 20:55:04,964 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681937704945, max timestamp=1681937704962Num uniques 859; 2023-04-19 20:55:04,964 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 20:55:04,978 DEBUG [Thread-1624] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:04,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:04,980 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:04,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:04,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:04,981 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:04,981 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:04,982 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:04,982 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:04,982 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:04,982 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:04,983 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:04,983 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:04,983 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:04,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:04,983 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:04,984 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:04,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:04,984 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:04,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:04,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:04,984 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:04,985 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:55:04,985 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:04,985 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:04,985 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:04,985 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:04,985 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:04,985 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:04,985 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. 2023-04-19 20:55:04,985 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. started... 2023-04-19 20:55:04,985 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. as already flushing 2023-04-19 20:55:04,985 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. 2023-04-19 20:55:04,985 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:04,985 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:04,985 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:04,985 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:04,985 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:04,989 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:04,989 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:04,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:04,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:04,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:04,990 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:04,990 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:04,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:04,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:04,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:04,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:04,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:04,991 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:04,991 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@554e240f[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:04,991 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:04,991 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:04,993 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:04,993 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:04,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:04,993 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:04,993 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:04,993 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:04,993 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:04,993 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:04,994 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:04,994 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:04,994 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:04,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:04,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:04,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:04,994 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:04,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:04,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:04,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:04,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:04,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:04,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:04,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:04,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:04,997 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:04,997 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:04,997 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:04,997 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:04,997 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:04,997 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:04,998 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:04,998 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:04,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:04,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:04,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:04,998 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:04,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:04,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:04,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:04,999 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:04,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:04,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:04,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:04,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:04,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:05,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:05,000 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:05,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:05,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:05,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:05,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:05,002 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:05,002 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:05,002 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:05,002 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:05,002 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:05,003 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:05,002 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:05,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:05,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:05,002 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:05,003 DEBUG [Thread-1624] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:05,003 DEBUG [Thread-1624] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:05,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:05,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:05,003 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:55:05,003 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:05,003 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:05,003 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:05,003 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:05,045 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:05,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 491 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34548 deadline: 1681937765042, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:05,103 DEBUG [Thread-1624] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:05,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:05,105 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:05,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:05,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:05,107 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:05,107 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:05,107 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:05,107 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:05,107 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:05,107 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:05,110 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:05,110 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:05,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:05,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:05,110 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:05,110 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:05,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:05,110 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:05,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:05,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:05,111 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:05,111 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:55:05,112 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:05,112 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:05,112 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:05,112 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:05,112 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:05,112 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:05,112 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. 2023-04-19 20:55:05,112 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. started... 2023-04-19 20:55:05,112 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. as already flushing 2023-04-19 20:55:05,112 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. 2023-04-19 20:55:05,112 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:05,112 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:05,112 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:05,112 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:05,112 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:05,114 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:05,114 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:05,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:05,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:05,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:05,114 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:05,114 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:05,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:05,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:05,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:05,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:05,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:05,116 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:05,116 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2acbe4a6[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:05,116 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:05,116 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:05,117 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:05,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:05,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:05,117 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:05,117 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:05,117 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:05,117 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:05,117 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:05,119 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:05,119 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:05,119 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:05,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:05,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:05,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:05,119 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:05,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:05,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:05,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:05,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:05,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:05,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:05,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:05,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:05,121 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:05,121 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:05,121 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:05,121 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:05,121 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:05,122 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:05,126 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:05,126 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:05,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:05,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:05,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:05,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:05,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:05,126 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:05,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:05,127 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:05,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:05,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:05,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:05,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:05,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:05,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:05,128 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:05,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:05,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:05,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:05,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:05,130 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:05,130 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:05,130 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:05,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:05,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:05,130 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:05,130 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:05,130 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:05,131 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:05,131 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:05,131 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:05,131 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:05,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:05,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:05,130 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:55:05,130 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:05,131 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:05,131 DEBUG [Thread-1624] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:05,131 DEBUG [Thread-1624] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:05,148 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:05,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 493 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34548 deadline: 1681937765146, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:05,231 DEBUG [Thread-1624] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:05,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:05,233 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:05,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:05,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:05,234 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:05,234 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:05,234 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:05,235 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:05,235 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:05,235 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:05,236 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:05,236 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:05,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:05,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:05,237 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:05,237 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:05,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:05,237 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:05,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:05,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:05,237 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:05,238 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:55:05,238 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:05,238 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:05,238 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:05,238 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:05,238 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:05,238 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:05,238 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. 2023-04-19 20:55:05,238 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. started... 2023-04-19 20:55:05,238 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. as already flushing 2023-04-19 20:55:05,238 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. 2023-04-19 20:55:05,238 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:05,238 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:05,238 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:05,238 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:05,239 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:05,240 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:05,240 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:05,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:05,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:05,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:05,240 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:05,240 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:05,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:05,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:05,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:05,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:05,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:05,242 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:05,242 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@432c67a[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:05,242 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:05,242 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:05,244 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:05,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:05,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:05,244 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:05,244 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:05,244 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:05,244 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:05,244 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:05,246 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:05,246 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:05,246 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:05,246 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:05,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:05,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:05,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:05,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:05,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:05,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:05,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:05,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:05,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:05,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:05,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:05,248 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:05,248 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:05,248 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:05,248 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:05,248 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:05,248 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:05,250 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:05,250 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:05,250 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:05,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:05,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:05,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:05,250 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:05,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:05,251 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:05,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:05,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:05,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:05,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:05,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:05,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:05,252 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:05,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:05,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:05,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:05,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:05,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:05,255 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:05,255 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:05,255 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:05,255 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:05,255 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:05,255 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:05,255 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:05,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:05,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:05,255 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:05,255 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:05,255 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:05,255 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:05,255 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:05,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:05,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:05,255 DEBUG [Thread-1624] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:05,255 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:55:05,256 DEBUG [Thread-1624] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:05,297 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=618 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/A/20351622b5dd4ec7aaa56882efb17c83 2023-04-19 20:55:05,307 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=618 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/B/0e033396ba7f4bd9a540fa0e50d96a6a 2023-04-19 20:55:05,352 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:05,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 495 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34548 deadline: 1681937765349, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:05,356 DEBUG [Thread-1624] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:05,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:05,358 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:05,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:05,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:05,359 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:05,359 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:05,359 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:05,359 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:05,359 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:05,359 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:05,362 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:05,362 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:05,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:05,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:05,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:05,362 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:05,362 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:05,362 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:05,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:05,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:05,363 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:05,363 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:55:05,363 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:05,363 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:05,363 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:05,363 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:05,363 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:05,364 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:05,364 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. 2023-04-19 20:55:05,364 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. started... 2023-04-19 20:55:05,364 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. as already flushing 2023-04-19 20:55:05,364 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. 2023-04-19 20:55:05,364 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:05,364 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:05,364 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:05,364 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:05,364 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:05,365 DEBUG [Thread-1629] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x681ace6c to 127.0.0.1:58736 2023-04-19 20:55:05,365 DEBUG [Thread-1629] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-19 20:55:05,366 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:05,366 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:05,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:05,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:05,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:05,367 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:05,367 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:05,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:05,367 DEBUG [Thread-1631] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2b4fa5d3 to 127.0.0.1:58736 2023-04-19 20:55:05,367 DEBUG [Thread-1631] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-19 20:55:05,367 DEBUG [Thread-1633] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x45de5ad8 to 127.0.0.1:58736 2023-04-19 20:55:05,368 DEBUG [Thread-1633] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-19 20:55:05,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:05,369 DEBUG [Thread-1625] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5e5b1c14 to 127.0.0.1:58736 2023-04-19 20:55:05,369 DEBUG [Thread-1625] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-19 20:55:05,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:05,371 DEBUG [Thread-1627] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4f29c1c4 to 127.0.0.1:58736 2023-04-19 20:55:05,371 DEBUG [Thread-1627] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-19 20:55:05,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:05,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:05,375 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:05,375 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a0a326c[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:05,375 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:05,375 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:05,377 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:05,377 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:05,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:05,377 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:05,377 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:05,377 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:05,377 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:05,377 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:05,378 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:05,378 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:05,378 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:05,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:05,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:05,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:05,378 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:05,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:05,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:05,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:05,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:05,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:05,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:05,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:05,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:05,380 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:05,380 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:05,381 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:05,381 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:05,381 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:05,381 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:05,382 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:05,382 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:05,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:05,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:05,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:05,382 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:05,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:05,382 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:05,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:05,383 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:05,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:05,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:05,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:05,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:05,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:05,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:05,383 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:05,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:05,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:05,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:05,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:05,385 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:05,385 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:05,385 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:05,385 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:05,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:05,385 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:05,385 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:05,386 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:05,385 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:05,385 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:05,386 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:05,386 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:05,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:05,386 DEBUG [Thread-1624] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:05,386 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:05,386 DEBUG [Thread-1624] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:05,386 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:05,386 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:05,486 DEBUG [Thread-1624] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:05,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:05,655 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:05,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 497 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34548 deadline: 1681937765655, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:05,721 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=618 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/C/d615878753f34172a33825f907b0294d 2023-04-19 20:55:05,724 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/A/20351622b5dd4ec7aaa56882efb17c83 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/20351622b5dd4ec7aaa56882efb17c83 2023-04-19 20:55:05,727 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/20351622b5dd4ec7aaa56882efb17c83, entries=200, sequenceid=618, filesize=14.3 K 2023-04-19 20:55:05,728 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/B/0e033396ba7f4bd9a540fa0e50d96a6a as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/0e033396ba7f4bd9a540fa0e50d96a6a 2023-04-19 20:55:05,731 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/0e033396ba7f4bd9a540fa0e50d96a6a, entries=150, sequenceid=618, filesize=11.9 K 2023-04-19 20:55:05,732 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/C/d615878753f34172a33825f907b0294d as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/d615878753f34172a33825f907b0294d 2023-04-19 20:55:05,735 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/d615878753f34172a33825f907b0294d, entries=150, sequenceid=618, filesize=11.9 K 2023-04-19 20:55:05,735 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 09bc544118dfd67701ac39d6da1a53f4 in 845ms, sequenceid=618, compaction requested=true 2023-04-19 20:55:05,735 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 09bc544118dfd67701ac39d6da1a53f4: 2023-04-19 20:55:05,736 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 20:55:05,736 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 8 store files, 0 compacting, 8 eligible, 16 blocking 2023-04-19 20:55:05,736 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 8 store files, 0 compacting, 8 eligible, 16 blocking 2023-04-19 20:55:05,736 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 20:55:05,736 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 20:55:05,737 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 8 files of size 100678 starting at candidate #0 after considering 21 permutations with 21 in ratio 2023-04-19 20:55:05,738 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 8 files of size 98238 starting at candidate #0 after considering 21 permutations with 21 in ratio 2023-04-19 20:55:05,738 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1912): 09bc544118dfd67701ac39d6da1a53f4/A is initiating minor compaction (all files) 2023-04-19 20:55:05,738 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1912): 09bc544118dfd67701ac39d6da1a53f4/B is initiating minor compaction (all files) 2023-04-19 20:55:05,738 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 09bc544118dfd67701ac39d6da1a53f4/A in TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. 2023-04-19 20:55:05,738 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 09bc544118dfd67701ac39d6da1a53f4/B in TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. 2023-04-19 20:55:05,738 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/369cec6160634e64a3883168ae3ee266, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/9990a212308145d1a151976c97e4c009, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/6fc4c2b04bbf4f3aabbe68e6e85d9bfd, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/c41b3bdc48f6425b837de01a022fcf01, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/df3865b5cdd74e4b920d3039de3ba6d5, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/c1be002333a7466388824e4d77b119e1, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/20d8c2d9c1af44bca0bdde146ba7402d, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/20351622b5dd4ec7aaa56882efb17c83] into tmpdir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp, totalSize=98.3 K 2023-04-19 20:55:05,738 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/154ffbdf7a8c492ca2d85d67a5172496, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/481b8141ee7a4f2fa36950f0326bc099, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/2813c1781ec84f0c8894113d55713faf, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/a79caa3698db471280ee302caf05e4ee, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/6475b28a544a423c91215ef654acdc1c, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/581204aa24eb48b5bc935992d8083a2a, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/2841a6cdafab4f7f9a8569decde21f5b, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/0e033396ba7f4bd9a540fa0e50d96a6a] into tmpdir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp, totalSize=95.9 K 2023-04-19 20:55:05,738 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 369cec6160634e64a3883168ae3ee266, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=464, earliestPutTs=1681937698091 2023-04-19 20:55:05,738 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 154ffbdf7a8c492ca2d85d67a5172496, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=464, earliestPutTs=1681937698091 2023-04-19 20:55:05,738 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 9990a212308145d1a151976c97e4c009, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=500, earliestPutTs=1681937699485 2023-04-19 20:55:05,738 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 481b8141ee7a4f2fa36950f0326bc099, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=500, earliestPutTs=1681937699485 2023-04-19 20:55:05,739 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 6fc4c2b04bbf4f3aabbe68e6e85d9bfd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=514, earliestPutTs=1681937700627 2023-04-19 20:55:05,739 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 2813c1781ec84f0c8894113d55713faf, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=514, earliestPutTs=1681937700627 2023-04-19 20:55:05,739 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting c41b3bdc48f6425b837de01a022fcf01, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=550, earliestPutTs=1681937701843 2023-04-19 20:55:05,739 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting a79caa3698db471280ee302caf05e4ee, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=550, earliestPutTs=1681937701843 2023-04-19 20:55:05,739 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting df3865b5cdd74e4b920d3039de3ba6d5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=561, earliestPutTs=1681937702980 2023-04-19 20:55:05,739 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 6475b28a544a423c91215ef654acdc1c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=561, earliestPutTs=1681937702980 2023-04-19 20:55:05,740 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting c1be002333a7466388824e4d77b119e1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=597, earliestPutTs=1681937703392 2023-04-19 20:55:05,740 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 581204aa24eb48b5bc935992d8083a2a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=597, earliestPutTs=1681937703392 2023-04-19 20:55:05,740 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 20d8c2d9c1af44bca0bdde146ba7402d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=608, earliestPutTs=1681937704549 2023-04-19 20:55:05,740 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 2841a6cdafab4f7f9a8569decde21f5b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=608, earliestPutTs=1681937704549 2023-04-19 20:55:05,740 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 20351622b5dd4ec7aaa56882efb17c83, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=618, earliestPutTs=1681937704874 2023-04-19 20:55:05,740 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 0e033396ba7f4bd9a540fa0e50d96a6a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=618, earliestPutTs=1681937704881 2023-04-19 20:55:05,752 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] throttle.PressureAwareThroughputController(145): 09bc544118dfd67701ac39d6da1a53f4#A#compaction#287 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 57.69 MB/second 2023-04-19 20:55:05,752 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] throttle.PressureAwareThroughputController(145): 09bc544118dfd67701ac39d6da1a53f4#B#compaction#288 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 57.69 MB/second 2023-04-19 20:55:05,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(9158): Flush requested on 09bc544118dfd67701ac39d6da1a53f4 2023-04-19 20:55:05,944 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 09bc544118dfd67701ac39d6da1a53f4 3/3 column families, dataSize=140.89 KB heapSize=405.61 KB 2023-04-19 20:55:05,944 DEBUG [Thread-1616] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x10b79c2c to 127.0.0.1:58736 2023-04-19 20:55:05,944 DEBUG [Thread-1616] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-19 20:55:05,944 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09bc544118dfd67701ac39d6da1a53f4, store=A 2023-04-19 20:55:05,944 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-19 20:55:05,944 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09bc544118dfd67701ac39d6da1a53f4, store=B 2023-04-19 20:55:05,944 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-19 20:55:05,945 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09bc544118dfd67701ac39d6da1a53f4, store=C 2023-04-19 20:55:05,945 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-19 20:55:06,023 DEBUG [Thread-1622] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6705d115 to 127.0.0.1:58736 2023-04-19 20:55:06,023 DEBUG [Thread-1622] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-19 20:55:06,159 DEBUG [Thread-1620] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3562c7bf to 127.0.0.1:58736 2023-04-19 20:55:06,159 DEBUG [Thread-1620] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-19 20:55:06,162 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/B/837d01d059b843bca857e74884ead9a6 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/837d01d059b843bca857e74884ead9a6 2023-04-19 20:55:06,162 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/A/bb48f52f5cd0493f9cbf8b3cf8310d5e as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/bb48f52f5cd0493f9cbf8b3cf8310d5e 2023-04-19 20:55:06,166 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 8 (all) file(s) in 09bc544118dfd67701ac39d6da1a53f4/A of 09bc544118dfd67701ac39d6da1a53f4 into bb48f52f5cd0493f9cbf8b3cf8310d5e(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 20:55:06,166 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1652): Completed compaction of 8 (all) file(s) in 09bc544118dfd67701ac39d6da1a53f4/B of 09bc544118dfd67701ac39d6da1a53f4 into 837d01d059b843bca857e74884ead9a6(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 20:55:06,166 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 09bc544118dfd67701ac39d6da1a53f4: 2023-04-19 20:55:06,166 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 09bc544118dfd67701ac39d6da1a53f4: 2023-04-19 20:55:06,166 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4., storeName=09bc544118dfd67701ac39d6da1a53f4/A, priority=8, startTime=1681937705736; duration=0sec 2023-04-19 20:55:06,166 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4., storeName=09bc544118dfd67701ac39d6da1a53f4/B, priority=8, startTime=1681937705736; duration=0sec 2023-04-19 20:55:06,166 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 20:55:06,166 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 20:55:06,166 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 8 store files, 0 compacting, 8 eligible, 16 blocking 2023-04-19 20:55:06,168 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 8 files of size 98204 starting at candidate #0 after considering 21 permutations with 21 in ratio 2023-04-19 20:55:06,168 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1912): 09bc544118dfd67701ac39d6da1a53f4/C is initiating minor compaction (all files) 2023-04-19 20:55:06,168 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 09bc544118dfd67701ac39d6da1a53f4/C in TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. 2023-04-19 20:55:06,168 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/08ccad336b694a10be95d9e4f06d5367, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/8fc10a4ad5d74787a8db2ef8422504f3, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/7d6045f912414592bad9403bb9de6176, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/29905e6bdd3e4ce2b9fe5182aaa67b9b, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/75086013405f483c86c594069f6275f6, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/9850070a77314694a6257864c0c092bf, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/889f355d12ed4a2b8fc58a102c8d1d91, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/d615878753f34172a33825f907b0294d] into tmpdir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp, totalSize=95.9 K 2023-04-19 20:55:06,168 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 08ccad336b694a10be95d9e4f06d5367, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=464, earliestPutTs=1681937698091 2023-04-19 20:55:06,169 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 8fc10a4ad5d74787a8db2ef8422504f3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=500, earliestPutTs=1681937699485 2023-04-19 20:55:06,169 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 7d6045f912414592bad9403bb9de6176, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=514, earliestPutTs=1681937700627 2023-04-19 20:55:06,169 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 29905e6bdd3e4ce2b9fe5182aaa67b9b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=550, earliestPutTs=1681937701843 2023-04-19 20:55:06,169 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 75086013405f483c86c594069f6275f6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=561, earliestPutTs=1681937702980 2023-04-19 20:55:06,170 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 9850070a77314694a6257864c0c092bf, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=597, earliestPutTs=1681937703392 2023-04-19 20:55:06,170 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 889f355d12ed4a2b8fc58a102c8d1d91, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=608, earliestPutTs=1681937704549 2023-04-19 20:55:06,170 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting d615878753f34172a33825f907b0294d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=618, earliestPutTs=1681937704881 2023-04-19 20:55:06,179 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] throttle.PressureAwareThroughputController(145): 09bc544118dfd67701ac39d6da1a53f4#C#compaction#290 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 57.69 MB/second 2023-04-19 20:55:06,264 DEBUG [Thread-1614] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x13544701 to 127.0.0.1:58736 2023-04-19 20:55:06,264 DEBUG [Thread-1614] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-19 20:55:06,266 DEBUG [Thread-1618] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5a5a81fa to 127.0.0.1:58736 2023-04-19 20:55:06,266 DEBUG [Thread-1618] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-19 20:55:06,266 INFO [Listener at localhost/34893] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-19 20:55:06,266 INFO [Listener at localhost/34893] hbase.AcidGuaranteesTestTool(394): wrote 43 2023-04-19 20:55:06,266 INFO [Listener at localhost/34893] hbase.AcidGuaranteesTestTool(394): wrote 50 2023-04-19 20:55:06,266 INFO [Listener at localhost/34893] hbase.AcidGuaranteesTestTool(394): wrote 77 2023-04-19 20:55:06,267 INFO [Listener at localhost/34893] hbase.AcidGuaranteesTestTool(394): wrote 372 2023-04-19 20:55:06,267 INFO [Listener at localhost/34893] hbase.AcidGuaranteesTestTool(394): wrote 24 2023-04-19 20:55:06,267 INFO [Listener at localhost/34893] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-19 20:55:06,267 INFO [Listener at localhost/34893] hbase.AcidGuaranteesTestTool(398): read 5121 2023-04-19 20:55:06,267 INFO [Listener at localhost/34893] hbase.AcidGuaranteesTestTool(398): read 5248 2023-04-19 20:55:06,267 INFO [Listener at localhost/34893] hbase.AcidGuaranteesTestTool(398): read 5049 2023-04-19 20:55:06,267 INFO [Listener at localhost/34893] hbase.AcidGuaranteesTestTool(398): read 5114 2023-04-19 20:55:06,267 INFO [Listener at localhost/34893] hbase.AcidGuaranteesTestTool(398): read 5236 2023-04-19 20:55:06,267 INFO [Listener at localhost/34893] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-19 20:55:06,267 INFO [Listener at localhost/34893] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-19 20:55:06,267 DEBUG [Listener at localhost/34893] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x53921272 to 127.0.0.1:58736 2023-04-19 20:55:06,267 DEBUG [Listener at localhost/34893] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-19 20:55:06,267 INFO [Listener at localhost/34893] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-19 20:55:06,268 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-19 20:55:06,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure2.ProcedureExecutor(1029): Stored pid=31, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-19 20:55:06,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-19 20:55:06,271 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681937706271"}]},"ts":"1681937706271"} 2023-04-19 20:55:06,272 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-19 20:55:06,273 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-19 20:55:06,273 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=32, ppid=31, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=09bc544118dfd67701ac39d6da1a53f4, UNASSIGN}] 2023-04-19 20:55:06,275 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=32, ppid=31, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=09bc544118dfd67701ac39d6da1a53f4, UNASSIGN 2023-04-19 20:55:06,275 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=32 updating hbase:meta row=09bc544118dfd67701ac39d6da1a53f4, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:06,275 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681937706275"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681937706275"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681937706275"}]},"ts":"1681937706275"} 2023-04-19 20:55:06,276 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=33, ppid=32, state=RUNNABLE; CloseRegionProcedure 09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971}] 2023-04-19 20:55:06,351 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=656 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/A/634480a3bc044d86a123811a58660c43 2023-04-19 20:55:06,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-19 20:55:06,428 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 09bc544118dfd67701ac39d6da1a53f4 2023-04-19 20:55:06,428 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 09bc544118dfd67701ac39d6da1a53f4, disabling compactions & flushes 2023-04-19 20:55:06,428 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1858): waiting for 1 compactions & cache flush to complete for region TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. 2023-04-19 20:55:06,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-19 20:55:06,587 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/C/7b8be17f569f44d7875af30cde92933e as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/7b8be17f569f44d7875af30cde92933e 2023-04-19 20:55:06,590 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 8 (all) file(s) in 09bc544118dfd67701ac39d6da1a53f4/C of 09bc544118dfd67701ac39d6da1a53f4 into 7b8be17f569f44d7875af30cde92933e(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 20:55:06,590 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 09bc544118dfd67701ac39d6da1a53f4: 2023-04-19 20:55:06,590 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1858): waiting for 0 compactions & cache flush to complete for region TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. 2023-04-19 20:55:06,591 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4., storeName=09bc544118dfd67701ac39d6da1a53f4/C, priority=8, startTime=1681937705736; duration=0sec 2023-04-19 20:55:06,591 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 20:55:06,761 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=656 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/B/c74952e3d521405eaa2054924cf90489 2023-04-19 20:55:06,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-19 20:55:07,171 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=656 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/C/b4a212ed28204c609e482ace4c6c3ba3 2023-04-19 20:55:07,175 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/A/634480a3bc044d86a123811a58660c43 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/634480a3bc044d86a123811a58660c43 2023-04-19 20:55:07,178 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/634480a3bc044d86a123811a58660c43, entries=150, sequenceid=656, filesize=11.9 K 2023-04-19 20:55:07,179 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/B/c74952e3d521405eaa2054924cf90489 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/c74952e3d521405eaa2054924cf90489 2023-04-19 20:55:07,182 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/c74952e3d521405eaa2054924cf90489, entries=150, sequenceid=656, filesize=11.9 K 2023-04-19 20:55:07,182 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/C/b4a212ed28204c609e482ace4c6c3ba3 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/b4a212ed28204c609e482ace4c6c3ba3 2023-04-19 20:55:07,186 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/b4a212ed28204c609e482ace4c6c3ba3, entries=150, sequenceid=656, filesize=11.9 K 2023-04-19 20:55:07,186 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~405.56 KB/415296, currentSize=26.84 KB/27480 for 09bc544118dfd67701ac39d6da1a53f4 in 1242ms, sequenceid=656, compaction requested=false 2023-04-19 20:55:07,186 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 09bc544118dfd67701ac39d6da1a53f4: 2023-04-19 20:55:07,186 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. 2023-04-19 20:55:07,186 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. 2023-04-19 20:55:07,186 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. after waiting 0 ms 2023-04-19 20:55:07,186 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. 2023-04-19 20:55:07,186 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 09bc544118dfd67701ac39d6da1a53f4 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-19 20:55:07,187 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09bc544118dfd67701ac39d6da1a53f4, store=A 2023-04-19 20:55:07,187 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:55:07,187 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09bc544118dfd67701ac39d6da1a53f4, store=B 2023-04-19 20:55:07,187 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:55:07,187 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09bc544118dfd67701ac39d6da1a53f4, store=C 2023-04-19 20:55:07,187 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:55:07,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-19 20:55:07,594 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=666 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/A/daaf22a673da41d4a0f8b7ac3cf493dd 2023-04-19 20:55:08,002 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=666 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/B/833b5e6ead594d41baacd1db9da888fa 2023-04-19 20:55:08,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-19 20:55:08,411 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=666 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/C/7b26db556f664f5eb2ae3d07ee3402cd 2023-04-19 20:55:08,414 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/A/daaf22a673da41d4a0f8b7ac3cf493dd as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/daaf22a673da41d4a0f8b7ac3cf493dd 2023-04-19 20:55:08,417 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/daaf22a673da41d4a0f8b7ac3cf493dd, entries=100, sequenceid=666, filesize=9.5 K 2023-04-19 20:55:08,418 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/B/833b5e6ead594d41baacd1db9da888fa as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/833b5e6ead594d41baacd1db9da888fa 2023-04-19 20:55:08,421 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/833b5e6ead594d41baacd1db9da888fa, entries=100, sequenceid=666, filesize=9.5 K 2023-04-19 20:55:08,422 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/.tmp/C/7b26db556f664f5eb2ae3d07ee3402cd as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/7b26db556f664f5eb2ae3d07ee3402cd 2023-04-19 20:55:08,425 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/7b26db556f664f5eb2ae3d07ee3402cd, entries=100, sequenceid=666, filesize=9.5 K 2023-04-19 20:55:08,425 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for 09bc544118dfd67701ac39d6da1a53f4 in 1239ms, sequenceid=666, compaction requested=true 2023-04-19 20:55:08,446 DEBUG [StoreCloser-TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/f6063291209d427bad7d0a0922ab5ac4, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/1420f10daa8a47d28ce9cad6985d6c44, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/c57b1ce9282947dc8326ed3b2ea0670e, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/2958104446344d619c27f6255da723e5, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/df24c00b109c45d99964454a1fbbdca2, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/dd84fac044094914b5f1ff40bea933de, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/a262a79346ba4331a4de0393726ca17c, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/bf46b37dae664e82b75afe78db6c2acb, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/1ba5c75941d74be1b8659e0b8fc6da94, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/5461affcf7804820a05692cd1a68292e, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/8cbad9651fdc4c84bedbb94942796c01, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/c5ad4e0b96d7436d9c57b2c16e44260a, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/fa86b1b7451848e2b27bd1f6a10d8015, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/80a13ac695664bdc9436659c7fda85f7, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/8fa5b2d10c1e4bbba458b1a74e069e2b, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/7dbaba8ccae443c0a92a61d498d00efd, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/fdd7f4955e3349349d002125a8a6eeec, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/dff48212e2c2468b9a91ef1ac8a770d0, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/6bb51a26606649e7b436816cc9f29277, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/b5ba1dc9236c477098de0284b97c2585, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/783ea13a8c224da8810e5073514b598d, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/369cec6160634e64a3883168ae3ee266, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/3bbe895fafb545aea7c806d7715d344b, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/9990a212308145d1a151976c97e4c009, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/6fc4c2b04bbf4f3aabbe68e6e85d9bfd, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/c41b3bdc48f6425b837de01a022fcf01, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/df3865b5cdd74e4b920d3039de3ba6d5, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/c1be002333a7466388824e4d77b119e1, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/20d8c2d9c1af44bca0bdde146ba7402d, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/20351622b5dd4ec7aaa56882efb17c83] to archive 2023-04-19 20:55:08,447 DEBUG [StoreCloser-TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-19 20:55:08,449 DEBUG [StoreCloser-TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/f6063291209d427bad7d0a0922ab5ac4 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/f6063291209d427bad7d0a0922ab5ac4 2023-04-19 20:55:08,450 DEBUG [StoreCloser-TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/1420f10daa8a47d28ce9cad6985d6c44 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/1420f10daa8a47d28ce9cad6985d6c44 2023-04-19 20:55:08,450 DEBUG [StoreCloser-TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/c57b1ce9282947dc8326ed3b2ea0670e to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/c57b1ce9282947dc8326ed3b2ea0670e 2023-04-19 20:55:08,451 DEBUG [StoreCloser-TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/2958104446344d619c27f6255da723e5 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/2958104446344d619c27f6255da723e5 2023-04-19 20:55:08,452 DEBUG [StoreCloser-TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/df24c00b109c45d99964454a1fbbdca2 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/df24c00b109c45d99964454a1fbbdca2 2023-04-19 20:55:08,453 DEBUG [StoreCloser-TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/dd84fac044094914b5f1ff40bea933de to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/dd84fac044094914b5f1ff40bea933de 2023-04-19 20:55:08,454 DEBUG [StoreCloser-TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/a262a79346ba4331a4de0393726ca17c to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/a262a79346ba4331a4de0393726ca17c 2023-04-19 20:55:08,454 DEBUG [StoreCloser-TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/bf46b37dae664e82b75afe78db6c2acb to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/bf46b37dae664e82b75afe78db6c2acb 2023-04-19 20:55:08,455 DEBUG [StoreCloser-TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/1ba5c75941d74be1b8659e0b8fc6da94 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/1ba5c75941d74be1b8659e0b8fc6da94 2023-04-19 20:55:08,456 DEBUG [StoreCloser-TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/5461affcf7804820a05692cd1a68292e to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/5461affcf7804820a05692cd1a68292e 2023-04-19 20:55:08,457 DEBUG [StoreCloser-TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/8cbad9651fdc4c84bedbb94942796c01 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/8cbad9651fdc4c84bedbb94942796c01 2023-04-19 20:55:08,458 DEBUG [StoreCloser-TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/c5ad4e0b96d7436d9c57b2c16e44260a to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/c5ad4e0b96d7436d9c57b2c16e44260a 2023-04-19 20:55:08,458 DEBUG [StoreCloser-TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/fa86b1b7451848e2b27bd1f6a10d8015 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/fa86b1b7451848e2b27bd1f6a10d8015 2023-04-19 20:55:08,459 DEBUG [StoreCloser-TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/80a13ac695664bdc9436659c7fda85f7 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/80a13ac695664bdc9436659c7fda85f7 2023-04-19 20:55:08,460 DEBUG [StoreCloser-TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/8fa5b2d10c1e4bbba458b1a74e069e2b to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/8fa5b2d10c1e4bbba458b1a74e069e2b 2023-04-19 20:55:08,461 DEBUG [StoreCloser-TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/7dbaba8ccae443c0a92a61d498d00efd to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/7dbaba8ccae443c0a92a61d498d00efd 2023-04-19 20:55:08,462 DEBUG [StoreCloser-TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/fdd7f4955e3349349d002125a8a6eeec to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/fdd7f4955e3349349d002125a8a6eeec 2023-04-19 20:55:08,463 DEBUG [StoreCloser-TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/dff48212e2c2468b9a91ef1ac8a770d0 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/dff48212e2c2468b9a91ef1ac8a770d0 2023-04-19 20:55:08,464 DEBUG [StoreCloser-TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/6bb51a26606649e7b436816cc9f29277 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/6bb51a26606649e7b436816cc9f29277 2023-04-19 20:55:08,465 DEBUG [StoreCloser-TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/b5ba1dc9236c477098de0284b97c2585 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/b5ba1dc9236c477098de0284b97c2585 2023-04-19 20:55:08,466 DEBUG [StoreCloser-TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/783ea13a8c224da8810e5073514b598d to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/783ea13a8c224da8810e5073514b598d 2023-04-19 20:55:08,466 DEBUG [StoreCloser-TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/369cec6160634e64a3883168ae3ee266 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/369cec6160634e64a3883168ae3ee266 2023-04-19 20:55:08,467 DEBUG [StoreCloser-TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/3bbe895fafb545aea7c806d7715d344b to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/3bbe895fafb545aea7c806d7715d344b 2023-04-19 20:55:08,468 DEBUG [StoreCloser-TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/9990a212308145d1a151976c97e4c009 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/9990a212308145d1a151976c97e4c009 2023-04-19 20:55:08,469 DEBUG [StoreCloser-TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/6fc4c2b04bbf4f3aabbe68e6e85d9bfd to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/6fc4c2b04bbf4f3aabbe68e6e85d9bfd 2023-04-19 20:55:08,470 DEBUG [StoreCloser-TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/c41b3bdc48f6425b837de01a022fcf01 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/c41b3bdc48f6425b837de01a022fcf01 2023-04-19 20:55:08,471 DEBUG [StoreCloser-TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/df3865b5cdd74e4b920d3039de3ba6d5 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/df3865b5cdd74e4b920d3039de3ba6d5 2023-04-19 20:55:08,472 DEBUG [StoreCloser-TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/c1be002333a7466388824e4d77b119e1 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/c1be002333a7466388824e4d77b119e1 2023-04-19 20:55:08,472 DEBUG [StoreCloser-TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/20d8c2d9c1af44bca0bdde146ba7402d to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/20d8c2d9c1af44bca0bdde146ba7402d 2023-04-19 20:55:08,473 DEBUG [StoreCloser-TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/20351622b5dd4ec7aaa56882efb17c83 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/20351622b5dd4ec7aaa56882efb17c83 2023-04-19 20:55:08,476 DEBUG [StoreCloser-TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/82f6f8fd358241138162f76c87217e99, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/1bb1be621f6e434ab49bef17e8960cfa, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/6dcfc11e83034a8dae373c2c552ab327, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/0492788529c84900b44105b6f81a53ad, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/d1c36bdb21ac4a6b8f682ba8ded71415, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/82eaedbf08294b58830dc03c11c92fd6, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/658221b251bc41e08cdddc0e79797404, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/bfcf02323c734c7f920ed57c840e7d1e, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/ca9c96d6334747d0b3e23833b43f22b2, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/3ea8fc98b3cd432d8fa8c22cc0602a52, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/47601069633c461a87e550937cb4a47a, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/c962658efe874b22852e9aa20182d5b7, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/38592d5aee7b47ad8d8413ea030a6c8f, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/26f2bb18a4f14358b03957a3f5ade3fe, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/076ea9fbd3224c9e9f9142dc8f61a8ee, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/8fb851af21914f5793e7128f1612a95b, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/3e0baffb4b364763b046329b1769b9d6, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/4893a404865c4f69aa7f8dbfb8bd3a08, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/dd9479e8bd094a7ead99cbc4ab9c9b1a, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/e1801e9ceb804253841b6fdc70000bcc, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/c13b49e4a2dd42109d7eeb8de2de4a6f, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/154ffbdf7a8c492ca2d85d67a5172496, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/bc12d65daff24a45aed8552f5eb61c2a, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/481b8141ee7a4f2fa36950f0326bc099, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/2813c1781ec84f0c8894113d55713faf, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/a79caa3698db471280ee302caf05e4ee, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/6475b28a544a423c91215ef654acdc1c, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/581204aa24eb48b5bc935992d8083a2a, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/2841a6cdafab4f7f9a8569decde21f5b, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/0e033396ba7f4bd9a540fa0e50d96a6a] to archive 2023-04-19 20:55:08,477 DEBUG [StoreCloser-TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-19 20:55:08,478 DEBUG [StoreCloser-TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/82f6f8fd358241138162f76c87217e99 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/82f6f8fd358241138162f76c87217e99 2023-04-19 20:55:08,479 DEBUG [StoreCloser-TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/1bb1be621f6e434ab49bef17e8960cfa to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/1bb1be621f6e434ab49bef17e8960cfa 2023-04-19 20:55:08,480 DEBUG [StoreCloser-TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/6dcfc11e83034a8dae373c2c552ab327 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/6dcfc11e83034a8dae373c2c552ab327 2023-04-19 20:55:08,481 DEBUG [StoreCloser-TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/0492788529c84900b44105b6f81a53ad to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/0492788529c84900b44105b6f81a53ad 2023-04-19 20:55:08,481 DEBUG [StoreCloser-TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/d1c36bdb21ac4a6b8f682ba8ded71415 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/d1c36bdb21ac4a6b8f682ba8ded71415 2023-04-19 20:55:08,482 DEBUG [StoreCloser-TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/82eaedbf08294b58830dc03c11c92fd6 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/82eaedbf08294b58830dc03c11c92fd6 2023-04-19 20:55:08,483 DEBUG [StoreCloser-TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/658221b251bc41e08cdddc0e79797404 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/658221b251bc41e08cdddc0e79797404 2023-04-19 20:55:08,484 DEBUG [StoreCloser-TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/bfcf02323c734c7f920ed57c840e7d1e to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/bfcf02323c734c7f920ed57c840e7d1e 2023-04-19 20:55:08,485 DEBUG [StoreCloser-TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/ca9c96d6334747d0b3e23833b43f22b2 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/ca9c96d6334747d0b3e23833b43f22b2 2023-04-19 20:55:08,485 DEBUG [StoreCloser-TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/3ea8fc98b3cd432d8fa8c22cc0602a52 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/3ea8fc98b3cd432d8fa8c22cc0602a52 2023-04-19 20:55:08,486 DEBUG [StoreCloser-TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/47601069633c461a87e550937cb4a47a to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/47601069633c461a87e550937cb4a47a 2023-04-19 20:55:08,487 DEBUG [StoreCloser-TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/c962658efe874b22852e9aa20182d5b7 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/c962658efe874b22852e9aa20182d5b7 2023-04-19 20:55:08,488 DEBUG [StoreCloser-TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/38592d5aee7b47ad8d8413ea030a6c8f to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/38592d5aee7b47ad8d8413ea030a6c8f 2023-04-19 20:55:08,489 DEBUG [StoreCloser-TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/26f2bb18a4f14358b03957a3f5ade3fe to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/26f2bb18a4f14358b03957a3f5ade3fe 2023-04-19 20:55:08,490 DEBUG [StoreCloser-TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/076ea9fbd3224c9e9f9142dc8f61a8ee to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/076ea9fbd3224c9e9f9142dc8f61a8ee 2023-04-19 20:55:08,490 DEBUG [StoreCloser-TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/8fb851af21914f5793e7128f1612a95b to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/8fb851af21914f5793e7128f1612a95b 2023-04-19 20:55:08,491 DEBUG [StoreCloser-TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/3e0baffb4b364763b046329b1769b9d6 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/3e0baffb4b364763b046329b1769b9d6 2023-04-19 20:55:08,492 DEBUG [StoreCloser-TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/4893a404865c4f69aa7f8dbfb8bd3a08 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/4893a404865c4f69aa7f8dbfb8bd3a08 2023-04-19 20:55:08,493 DEBUG [StoreCloser-TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/dd9479e8bd094a7ead99cbc4ab9c9b1a to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/dd9479e8bd094a7ead99cbc4ab9c9b1a 2023-04-19 20:55:08,494 DEBUG [StoreCloser-TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/e1801e9ceb804253841b6fdc70000bcc to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/e1801e9ceb804253841b6fdc70000bcc 2023-04-19 20:55:08,494 DEBUG [StoreCloser-TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/c13b49e4a2dd42109d7eeb8de2de4a6f to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/c13b49e4a2dd42109d7eeb8de2de4a6f 2023-04-19 20:55:08,495 DEBUG [StoreCloser-TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/154ffbdf7a8c492ca2d85d67a5172496 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/154ffbdf7a8c492ca2d85d67a5172496 2023-04-19 20:55:08,496 DEBUG [StoreCloser-TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/bc12d65daff24a45aed8552f5eb61c2a to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/bc12d65daff24a45aed8552f5eb61c2a 2023-04-19 20:55:08,497 DEBUG [StoreCloser-TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/481b8141ee7a4f2fa36950f0326bc099 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/481b8141ee7a4f2fa36950f0326bc099 2023-04-19 20:55:08,497 DEBUG [StoreCloser-TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/2813c1781ec84f0c8894113d55713faf to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/2813c1781ec84f0c8894113d55713faf 2023-04-19 20:55:08,498 DEBUG [StoreCloser-TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/a79caa3698db471280ee302caf05e4ee to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/a79caa3698db471280ee302caf05e4ee 2023-04-19 20:55:08,499 DEBUG [StoreCloser-TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/6475b28a544a423c91215ef654acdc1c to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/6475b28a544a423c91215ef654acdc1c 2023-04-19 20:55:08,500 DEBUG [StoreCloser-TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/581204aa24eb48b5bc935992d8083a2a to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/581204aa24eb48b5bc935992d8083a2a 2023-04-19 20:55:08,501 DEBUG [StoreCloser-TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/2841a6cdafab4f7f9a8569decde21f5b to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/2841a6cdafab4f7f9a8569decde21f5b 2023-04-19 20:55:08,501 DEBUG [StoreCloser-TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/0e033396ba7f4bd9a540fa0e50d96a6a to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/0e033396ba7f4bd9a540fa0e50d96a6a 2023-04-19 20:55:08,504 DEBUG [StoreCloser-TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/e8344ec11af845dc877d43c20f89f903, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/99e8bfc6626a4015a341c31dce1ebbb9, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/94abbcc9eebe4d6f9e908f6c71835d25, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/17013f7f119840da8a2a16ea307d7997, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/56b9a028ace1448ca126392a183aa92e, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/7a7a84c1f3a148d18a3938f90ea1a0fd, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/a97c730a972f46ab8a1ee332e28dba03, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/7c0d9d7106b94bd58ed78a9afd39c515, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/ba02cc56583a4873953ca419c5714aeb, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/850542cc5c8b451281c3ade24602a9e8, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/a0fc09181c9547ad93ebdc3173541134, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/4490115a7b484a308c8ee5e9b419118a, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/3faa2fe5bd2f4bf6934b5ea37774aaf5, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/74506f098f154c4b8d5b42249e4a9a0a, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/082774a643da40d09836c6b732c57cfb, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/7d59300d429f48e5b6a630e2a0c20dba, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/f1b48a2afa87427883ae753f87ddcfa6, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/cd25a11463aa4b3ab93befbd95930d81, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/bbcbcfd1299c4b9ea452dbbc0694a752, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/6816143ad73140c7bae602854de528b2, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/08ccad336b694a10be95d9e4f06d5367, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/4e2ede360c804aaca211219533c9a658, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/8fc10a4ad5d74787a8db2ef8422504f3, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/7d6045f912414592bad9403bb9de6176, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/29905e6bdd3e4ce2b9fe5182aaa67b9b, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/75086013405f483c86c594069f6275f6, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/9850070a77314694a6257864c0c092bf, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/889f355d12ed4a2b8fc58a102c8d1d91, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/d615878753f34172a33825f907b0294d] to archive 2023-04-19 20:55:08,505 DEBUG [StoreCloser-TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-19 20:55:08,506 DEBUG [StoreCloser-TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/e8344ec11af845dc877d43c20f89f903 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/e8344ec11af845dc877d43c20f89f903 2023-04-19 20:55:08,507 DEBUG [StoreCloser-TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/99e8bfc6626a4015a341c31dce1ebbb9 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/99e8bfc6626a4015a341c31dce1ebbb9 2023-04-19 20:55:08,508 DEBUG [StoreCloser-TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/94abbcc9eebe4d6f9e908f6c71835d25 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/94abbcc9eebe4d6f9e908f6c71835d25 2023-04-19 20:55:08,509 DEBUG [StoreCloser-TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/17013f7f119840da8a2a16ea307d7997 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/17013f7f119840da8a2a16ea307d7997 2023-04-19 20:55:08,511 DEBUG [StoreCloser-TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/56b9a028ace1448ca126392a183aa92e to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/56b9a028ace1448ca126392a183aa92e 2023-04-19 20:55:08,512 DEBUG [StoreCloser-TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/7a7a84c1f3a148d18a3938f90ea1a0fd to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/7a7a84c1f3a148d18a3938f90ea1a0fd 2023-04-19 20:55:08,513 DEBUG [StoreCloser-TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/a97c730a972f46ab8a1ee332e28dba03 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/a97c730a972f46ab8a1ee332e28dba03 2023-04-19 20:55:08,514 DEBUG [StoreCloser-TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/7c0d9d7106b94bd58ed78a9afd39c515 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/7c0d9d7106b94bd58ed78a9afd39c515 2023-04-19 20:55:08,515 DEBUG [StoreCloser-TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/ba02cc56583a4873953ca419c5714aeb to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/ba02cc56583a4873953ca419c5714aeb 2023-04-19 20:55:08,516 DEBUG [StoreCloser-TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/850542cc5c8b451281c3ade24602a9e8 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/850542cc5c8b451281c3ade24602a9e8 2023-04-19 20:55:08,517 DEBUG [StoreCloser-TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/a0fc09181c9547ad93ebdc3173541134 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/a0fc09181c9547ad93ebdc3173541134 2023-04-19 20:55:08,518 DEBUG [StoreCloser-TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/4490115a7b484a308c8ee5e9b419118a to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/4490115a7b484a308c8ee5e9b419118a 2023-04-19 20:55:08,519 DEBUG [StoreCloser-TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/3faa2fe5bd2f4bf6934b5ea37774aaf5 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/3faa2fe5bd2f4bf6934b5ea37774aaf5 2023-04-19 20:55:08,521 DEBUG [StoreCloser-TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/74506f098f154c4b8d5b42249e4a9a0a to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/74506f098f154c4b8d5b42249e4a9a0a 2023-04-19 20:55:08,522 DEBUG [StoreCloser-TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/082774a643da40d09836c6b732c57cfb to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/082774a643da40d09836c6b732c57cfb 2023-04-19 20:55:08,523 DEBUG [StoreCloser-TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/7d59300d429f48e5b6a630e2a0c20dba to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/7d59300d429f48e5b6a630e2a0c20dba 2023-04-19 20:55:08,524 DEBUG [StoreCloser-TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/f1b48a2afa87427883ae753f87ddcfa6 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/f1b48a2afa87427883ae753f87ddcfa6 2023-04-19 20:55:08,525 DEBUG [StoreCloser-TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/cd25a11463aa4b3ab93befbd95930d81 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/cd25a11463aa4b3ab93befbd95930d81 2023-04-19 20:55:08,527 DEBUG [StoreCloser-TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/bbcbcfd1299c4b9ea452dbbc0694a752 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/bbcbcfd1299c4b9ea452dbbc0694a752 2023-04-19 20:55:08,528 DEBUG [StoreCloser-TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/6816143ad73140c7bae602854de528b2 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/6816143ad73140c7bae602854de528b2 2023-04-19 20:55:08,529 DEBUG [StoreCloser-TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/08ccad336b694a10be95d9e4f06d5367 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/08ccad336b694a10be95d9e4f06d5367 2023-04-19 20:55:08,531 DEBUG [StoreCloser-TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/4e2ede360c804aaca211219533c9a658 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/4e2ede360c804aaca211219533c9a658 2023-04-19 20:55:08,532 DEBUG [StoreCloser-TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/8fc10a4ad5d74787a8db2ef8422504f3 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/8fc10a4ad5d74787a8db2ef8422504f3 2023-04-19 20:55:08,534 DEBUG [StoreCloser-TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/7d6045f912414592bad9403bb9de6176 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/7d6045f912414592bad9403bb9de6176 2023-04-19 20:55:08,536 DEBUG [StoreCloser-TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/29905e6bdd3e4ce2b9fe5182aaa67b9b to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/29905e6bdd3e4ce2b9fe5182aaa67b9b 2023-04-19 20:55:08,537 DEBUG [StoreCloser-TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/75086013405f483c86c594069f6275f6 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/75086013405f483c86c594069f6275f6 2023-04-19 20:55:08,538 DEBUG [StoreCloser-TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/9850070a77314694a6257864c0c092bf to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/9850070a77314694a6257864c0c092bf 2023-04-19 20:55:08,539 DEBUG [StoreCloser-TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/889f355d12ed4a2b8fc58a102c8d1d91 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/889f355d12ed4a2b8fc58a102c8d1d91 2023-04-19 20:55:08,540 DEBUG [StoreCloser-TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/d615878753f34172a33825f907b0294d to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/d615878753f34172a33825f907b0294d 2023-04-19 20:55:08,545 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/recovered.edits/669.seqid, newMaxSeqId=669, maxSeqId=1 2023-04-19 20:55:08,546 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4. 2023-04-19 20:55:08,546 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 09bc544118dfd67701ac39d6da1a53f4: 2023-04-19 20:55:08,547 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 09bc544118dfd67701ac39d6da1a53f4 2023-04-19 20:55:08,548 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=32 updating hbase:meta row=09bc544118dfd67701ac39d6da1a53f4, regionState=CLOSED 2023-04-19 20:55:08,548 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681937708548"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681937708548"}]},"ts":"1681937708548"} 2023-04-19 20:55:08,550 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=33, resume processing ppid=32 2023-04-19 20:55:08,550 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=33, ppid=32, state=SUCCESS; CloseRegionProcedure 09bc544118dfd67701ac39d6da1a53f4, server=jenkins-hbase4.apache.org,35247,1681937616971 in 2.2730 sec 2023-04-19 20:55:08,551 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=32, resume processing ppid=31 2023-04-19 20:55:08,551 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=32, ppid=31, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=09bc544118dfd67701ac39d6da1a53f4, UNASSIGN in 2.2770 sec 2023-04-19 20:55:08,551 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681937708551"}]},"ts":"1681937708551"} 2023-04-19 20:55:08,552 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-19 20:55:08,554 INFO [PEWorker-4] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-19 20:55:08,555 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=31, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 2.2860 sec 2023-04-19 20:55:10,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-19 20:55:10,376 INFO [Listener at localhost/34893] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 31 completed 2023-04-19 20:55:10,376 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-19 20:55:10,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure2.ProcedureExecutor(1029): Stored pid=34, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-19 20:55:10,378 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=34, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-19 20:55:10,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1227): Checking to see if procedure is done pid=34 2023-04-19 20:55:10,379 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=34, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-19 20:55:10,383 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/.tmp/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4 2023-04-19 20:55:10,385 DEBUG [HFileArchiver-7] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/.tmp/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A, FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/.tmp/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B, FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/.tmp/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C, FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/.tmp/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/recovered.edits] 2023-04-19 20:55:10,387 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/.tmp/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/634480a3bc044d86a123811a58660c43 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/634480a3bc044d86a123811a58660c43 2023-04-19 20:55:10,388 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/.tmp/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/bb48f52f5cd0493f9cbf8b3cf8310d5e to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/bb48f52f5cd0493f9cbf8b3cf8310d5e 2023-04-19 20:55:10,389 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/.tmp/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/daaf22a673da41d4a0f8b7ac3cf493dd to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/A/daaf22a673da41d4a0f8b7ac3cf493dd 2023-04-19 20:55:10,391 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/.tmp/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/833b5e6ead594d41baacd1db9da888fa to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/833b5e6ead594d41baacd1db9da888fa 2023-04-19 20:55:10,392 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/.tmp/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/837d01d059b843bca857e74884ead9a6 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/837d01d059b843bca857e74884ead9a6 2023-04-19 20:55:10,393 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/.tmp/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/c74952e3d521405eaa2054924cf90489 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/B/c74952e3d521405eaa2054924cf90489 2023-04-19 20:55:10,395 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/.tmp/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/7b26db556f664f5eb2ae3d07ee3402cd to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/7b26db556f664f5eb2ae3d07ee3402cd 2023-04-19 20:55:10,396 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/.tmp/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/7b8be17f569f44d7875af30cde92933e to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/7b8be17f569f44d7875af30cde92933e 2023-04-19 20:55:10,397 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/.tmp/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/b4a212ed28204c609e482ace4c6c3ba3 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/C/b4a212ed28204c609e482ace4c6c3ba3 2023-04-19 20:55:10,399 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/.tmp/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/recovered.edits/669.seqid to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4/recovered.edits/669.seqid 2023-04-19 20:55:10,399 DEBUG [HFileArchiver-7] backup.HFileArchiver(596): Deleted hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/.tmp/data/default/TestAcidGuarantees/09bc544118dfd67701ac39d6da1a53f4 2023-04-19 20:55:10,399 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-19 20:55:10,401 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=34, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-19 20:55:10,402 WARN [PEWorker-5] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-19 20:55:10,404 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-19 20:55:10,405 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=34, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-19 20:55:10,405 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-19 20:55:10,405 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1681937710405"}]},"ts":"9223372036854775807"} 2023-04-19 20:55:10,406 INFO [PEWorker-5] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-19 20:55:10,406 DEBUG [PEWorker-5] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 09bc544118dfd67701ac39d6da1a53f4, NAME => 'TestAcidGuarantees,,1681937683196.09bc544118dfd67701ac39d6da1a53f4.', STARTKEY => '', ENDKEY => ''}] 2023-04-19 20:55:10,406 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-19 20:55:10,406 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1681937710406"}]},"ts":"9223372036854775807"} 2023-04-19 20:55:10,407 INFO [PEWorker-5] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-19 20:55:10,409 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(130): Finished pid=34, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-19 20:55:10,410 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=34, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 33 msec 2023-04-19 20:55:10,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1227): Checking to see if procedure is done pid=34 2023-04-19 20:55:10,480 INFO [Listener at localhost/34893] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 34 completed 2023-04-19 20:55:10,491 INFO [Listener at localhost/34893] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testGetAtomicity Thread=228 (was 227) - Thread LEAK? -, OpenFileDescriptor=557 (was 583), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=331 (was 348), ProcessCount=172 (was 169) - ProcessCount LEAK? -, AvailableMemoryMB=5509 (was 5561) 2023-04-19 20:55:10,503 INFO [Listener at localhost/34893] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobScanAtomicity Thread=228, OpenFileDescriptor=557, MaxFileDescriptor=60000, SystemLoadAverage=331, ProcessCount=172, AvailableMemoryMB=5509 2023-04-19 20:55:10,504 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-19 20:55:10,505 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-19 20:55:10,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure2.ProcedureExecutor(1029): Stored pid=35, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-19 20:55:10,507 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-19 20:55:10,507 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 35 2023-04-19 20:55:10,508 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-19 20:55:10,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-19 20:55:10,509 DEBUG [HFileArchiver-8] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/.tmp/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc 2023-04-19 20:55:10,509 DEBUG [HFileArchiver-8] backup.HFileArchiver(153): Directory hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/.tmp/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc empty. 2023-04-19 20:55:10,509 DEBUG [HFileArchiver-8] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/.tmp/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc 2023-04-19 20:55:10,510 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-19 20:55:10,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-19 20:55:10,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-19 20:55:10,916 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-19 20:55:10,918 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 5067d98ca5f8d3760c5d0eb2b8bb06dc, NAME => 'TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/.tmp 2023-04-19 20:55:11,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-19 20:55:11,324 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-19 20:55:11,324 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 5067d98ca5f8d3760c5d0eb2b8bb06dc, disabling compactions & flushes 2023-04-19 20:55:11,324 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:11,324 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:11,324 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. after waiting 0 ms 2023-04-19 20:55:11,324 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:11,324 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:11,324 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 5067d98ca5f8d3760c5d0eb2b8bb06dc: 2023-04-19 20:55:11,326 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-19 20:55:11,327 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681937711327"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681937711327"}]},"ts":"1681937711327"} 2023-04-19 20:55:11,328 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-19 20:55:11,328 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-19 20:55:11,328 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681937711328"}]},"ts":"1681937711328"} 2023-04-19 20:55:11,329 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-19 20:55:11,332 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=5067d98ca5f8d3760c5d0eb2b8bb06dc, ASSIGN}] 2023-04-19 20:55:11,334 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=5067d98ca5f8d3760c5d0eb2b8bb06dc, ASSIGN 2023-04-19 20:55:11,334 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=5067d98ca5f8d3760c5d0eb2b8bb06dc, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,35247,1681937616971; forceNewPlan=false, retain=false 2023-04-19 20:55:11,485 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=36 updating hbase:meta row=5067d98ca5f8d3760c5d0eb2b8bb06dc, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:11,486 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681937711485"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681937711485"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681937711485"}]},"ts":"1681937711485"} 2023-04-19 20:55:11,487 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=37, ppid=36, state=RUNNABLE; OpenRegionProcedure 5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971}] 2023-04-19 20:55:11,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-19 20:55:11,642 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:11,642 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 5067d98ca5f8d3760c5d0eb2b8bb06dc, NAME => 'TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc.', STARTKEY => '', ENDKEY => ''} 2023-04-19 20:55:11,642 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 5067d98ca5f8d3760c5d0eb2b8bb06dc 2023-04-19 20:55:11,642 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-19 20:55:11,642 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 5067d98ca5f8d3760c5d0eb2b8bb06dc 2023-04-19 20:55:11,642 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 5067d98ca5f8d3760c5d0eb2b8bb06dc 2023-04-19 20:55:11,643 INFO [StoreOpener-5067d98ca5f8d3760c5d0eb2b8bb06dc-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 5067d98ca5f8d3760c5d0eb2b8bb06dc 2023-04-19 20:55:11,644 DEBUG [StoreOpener-5067d98ca5f8d3760c5d0eb2b8bb06dc-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A 2023-04-19 20:55:11,644 DEBUG [StoreOpener-5067d98ca5f8d3760c5d0eb2b8bb06dc-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A 2023-04-19 20:55:11,645 INFO [StoreOpener-5067d98ca5f8d3760c5d0eb2b8bb06dc-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-19 20:55:11,645 INFO [StoreOpener-5067d98ca5f8d3760c5d0eb2b8bb06dc-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 5067d98ca5f8d3760c5d0eb2b8bb06dc columnFamilyName A 2023-04-19 20:55:11,645 INFO [StoreOpener-5067d98ca5f8d3760c5d0eb2b8bb06dc-1] regionserver.HStore(310): Store=5067d98ca5f8d3760c5d0eb2b8bb06dc/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-19 20:55:11,645 INFO [StoreOpener-5067d98ca5f8d3760c5d0eb2b8bb06dc-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 5067d98ca5f8d3760c5d0eb2b8bb06dc 2023-04-19 20:55:11,646 DEBUG [StoreOpener-5067d98ca5f8d3760c5d0eb2b8bb06dc-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B 2023-04-19 20:55:11,646 DEBUG [StoreOpener-5067d98ca5f8d3760c5d0eb2b8bb06dc-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B 2023-04-19 20:55:11,646 INFO [StoreOpener-5067d98ca5f8d3760c5d0eb2b8bb06dc-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-19 20:55:11,647 INFO [StoreOpener-5067d98ca5f8d3760c5d0eb2b8bb06dc-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 5067d98ca5f8d3760c5d0eb2b8bb06dc columnFamilyName B 2023-04-19 20:55:11,647 INFO [StoreOpener-5067d98ca5f8d3760c5d0eb2b8bb06dc-1] regionserver.HStore(310): Store=5067d98ca5f8d3760c5d0eb2b8bb06dc/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-19 20:55:11,647 INFO [StoreOpener-5067d98ca5f8d3760c5d0eb2b8bb06dc-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 5067d98ca5f8d3760c5d0eb2b8bb06dc 2023-04-19 20:55:11,648 DEBUG [StoreOpener-5067d98ca5f8d3760c5d0eb2b8bb06dc-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C 2023-04-19 20:55:11,648 DEBUG [StoreOpener-5067d98ca5f8d3760c5d0eb2b8bb06dc-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C 2023-04-19 20:55:11,648 INFO [StoreOpener-5067d98ca5f8d3760c5d0eb2b8bb06dc-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-19 20:55:11,648 INFO [StoreOpener-5067d98ca5f8d3760c5d0eb2b8bb06dc-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 5067d98ca5f8d3760c5d0eb2b8bb06dc columnFamilyName C 2023-04-19 20:55:11,649 INFO [StoreOpener-5067d98ca5f8d3760c5d0eb2b8bb06dc-1] regionserver.HStore(310): Store=5067d98ca5f8d3760c5d0eb2b8bb06dc/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-19 20:55:11,649 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:11,649 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc 2023-04-19 20:55:11,650 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc 2023-04-19 20:55:11,651 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-19 20:55:11,652 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 5067d98ca5f8d3760c5d0eb2b8bb06dc 2023-04-19 20:55:11,654 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-19 20:55:11,654 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 5067d98ca5f8d3760c5d0eb2b8bb06dc; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=62737862, jitterRate=-0.0651330053806305}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-19 20:55:11,654 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 5067d98ca5f8d3760c5d0eb2b8bb06dc: 2023-04-19 20:55:11,655 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc., pid=37, masterSystemTime=1681937711638 2023-04-19 20:55:11,656 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:11,656 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:11,657 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=36 updating hbase:meta row=5067d98ca5f8d3760c5d0eb2b8bb06dc, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:11,657 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681937711656"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681937711656"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681937711656"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681937711656"}]},"ts":"1681937711656"} 2023-04-19 20:55:11,658 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=37, resume processing ppid=36 2023-04-19 20:55:11,658 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=37, ppid=36, state=SUCCESS; OpenRegionProcedure 5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 in 171 msec 2023-04-19 20:55:11,659 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=36, resume processing ppid=35 2023-04-19 20:55:11,660 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=36, ppid=35, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=5067d98ca5f8d3760c5d0eb2b8bb06dc, ASSIGN in 326 msec 2023-04-19 20:55:11,660 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-19 20:55:11,660 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681937711660"}]},"ts":"1681937711660"} 2023-04-19 20:55:11,661 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-19 20:55:11,662 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-19 20:55:11,663 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=35, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1570 sec 2023-04-19 20:55:12,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-19 20:55:12,612 INFO [Listener at localhost/34893] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 35 completed 2023-04-19 20:55:12,613 DEBUG [Listener at localhost/34893] zookeeper.ReadOnlyZKClient(139): Connect 0x118e131f to 127.0.0.1:58736 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-19 20:55:12,618 DEBUG [Listener at localhost/34893] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1cee66b5, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-19 20:55:12,619 DEBUG [Listener at localhost/34893] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-19 20:55:12,621 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:51224, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-19 20:55:12,622 DEBUG [Listener at localhost/34893] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-19 20:55:12,623 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46504, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-19 20:55:12,625 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-19 20:55:12,625 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.HMaster$12(2454): Client=jenkins//172.31.14.131 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-19 20:55:12,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure2.ProcedureExecutor(1029): Stored pid=38, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-04-19 20:55:13,034 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-19 20:55:13,035 DEBUG [PEWorker-3] util.FSTableDescriptors(519): Deleted hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-19 20:55:13,035 INFO [PEWorker-3] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-19 20:55:13,037 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=39, ppid=38, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-04-19 20:55:13,039 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=5067d98ca5f8d3760c5d0eb2b8bb06dc, REOPEN/MOVE}] 2023-04-19 20:55:13,040 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=5067d98ca5f8d3760c5d0eb2b8bb06dc, REOPEN/MOVE 2023-04-19 20:55:13,040 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=5067d98ca5f8d3760c5d0eb2b8bb06dc, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:13,041 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681937713040"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681937713040"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681937713040"}]},"ts":"1681937713040"} 2023-04-19 20:55:13,041 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=41, ppid=40, state=RUNNABLE; CloseRegionProcedure 5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971}] 2023-04-19 20:55:13,193 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 5067d98ca5f8d3760c5d0eb2b8bb06dc 2023-04-19 20:55:13,194 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 5067d98ca5f8d3760c5d0eb2b8bb06dc, disabling compactions & flushes 2023-04-19 20:55:13,194 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:13,194 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:13,194 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. after waiting 0 ms 2023-04-19 20:55:13,194 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:13,197 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-04-19 20:55:13,197 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:13,197 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 5067d98ca5f8d3760c5d0eb2b8bb06dc: 2023-04-19 20:55:13,198 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(3509): Not adding moved region record: 5067d98ca5f8d3760c5d0eb2b8bb06dc to self. 2023-04-19 20:55:13,199 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 5067d98ca5f8d3760c5d0eb2b8bb06dc 2023-04-19 20:55:13,199 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=5067d98ca5f8d3760c5d0eb2b8bb06dc, regionState=CLOSED 2023-04-19 20:55:13,199 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681937713199"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681937713199"}]},"ts":"1681937713199"} 2023-04-19 20:55:13,201 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=41, resume processing ppid=40 2023-04-19 20:55:13,201 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=41, ppid=40, state=SUCCESS; CloseRegionProcedure 5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 in 159 msec 2023-04-19 20:55:13,202 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=5067d98ca5f8d3760c5d0eb2b8bb06dc, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase4.apache.org,35247,1681937616971; forceNewPlan=false, retain=true 2023-04-19 20:55:13,352 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=5067d98ca5f8d3760c5d0eb2b8bb06dc, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:13,352 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681937713352"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681937713352"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681937713352"}]},"ts":"1681937713352"} 2023-04-19 20:55:13,354 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=42, ppid=40, state=RUNNABLE; OpenRegionProcedure 5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971}] 2023-04-19 20:55:13,508 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:13,508 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 5067d98ca5f8d3760c5d0eb2b8bb06dc, NAME => 'TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc.', STARTKEY => '', ENDKEY => ''} 2023-04-19 20:55:13,508 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 5067d98ca5f8d3760c5d0eb2b8bb06dc 2023-04-19 20:55:13,508 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-19 20:55:13,509 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 5067d98ca5f8d3760c5d0eb2b8bb06dc 2023-04-19 20:55:13,509 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 5067d98ca5f8d3760c5d0eb2b8bb06dc 2023-04-19 20:55:13,509 INFO [StoreOpener-5067d98ca5f8d3760c5d0eb2b8bb06dc-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 5067d98ca5f8d3760c5d0eb2b8bb06dc 2023-04-19 20:55:13,510 DEBUG [StoreOpener-5067d98ca5f8d3760c5d0eb2b8bb06dc-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A 2023-04-19 20:55:13,510 DEBUG [StoreOpener-5067d98ca5f8d3760c5d0eb2b8bb06dc-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A 2023-04-19 20:55:13,511 INFO [StoreOpener-5067d98ca5f8d3760c5d0eb2b8bb06dc-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-19 20:55:13,511 INFO [StoreOpener-5067d98ca5f8d3760c5d0eb2b8bb06dc-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 5067d98ca5f8d3760c5d0eb2b8bb06dc columnFamilyName A 2023-04-19 20:55:13,512 INFO [StoreOpener-5067d98ca5f8d3760c5d0eb2b8bb06dc-1] regionserver.HStore(310): Store=5067d98ca5f8d3760c5d0eb2b8bb06dc/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-19 20:55:13,512 INFO [StoreOpener-5067d98ca5f8d3760c5d0eb2b8bb06dc-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 5067d98ca5f8d3760c5d0eb2b8bb06dc 2023-04-19 20:55:13,513 DEBUG [StoreOpener-5067d98ca5f8d3760c5d0eb2b8bb06dc-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B 2023-04-19 20:55:13,513 DEBUG [StoreOpener-5067d98ca5f8d3760c5d0eb2b8bb06dc-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B 2023-04-19 20:55:13,513 INFO [StoreOpener-5067d98ca5f8d3760c5d0eb2b8bb06dc-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-19 20:55:13,513 INFO [StoreOpener-5067d98ca5f8d3760c5d0eb2b8bb06dc-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 5067d98ca5f8d3760c5d0eb2b8bb06dc columnFamilyName B 2023-04-19 20:55:13,513 INFO [StoreOpener-5067d98ca5f8d3760c5d0eb2b8bb06dc-1] regionserver.HStore(310): Store=5067d98ca5f8d3760c5d0eb2b8bb06dc/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-19 20:55:13,514 INFO [StoreOpener-5067d98ca5f8d3760c5d0eb2b8bb06dc-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 5067d98ca5f8d3760c5d0eb2b8bb06dc 2023-04-19 20:55:13,514 DEBUG [StoreOpener-5067d98ca5f8d3760c5d0eb2b8bb06dc-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C 2023-04-19 20:55:13,514 DEBUG [StoreOpener-5067d98ca5f8d3760c5d0eb2b8bb06dc-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C 2023-04-19 20:55:13,514 INFO [StoreOpener-5067d98ca5f8d3760c5d0eb2b8bb06dc-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-19 20:55:13,514 INFO [StoreOpener-5067d98ca5f8d3760c5d0eb2b8bb06dc-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 5067d98ca5f8d3760c5d0eb2b8bb06dc columnFamilyName C 2023-04-19 20:55:13,515 INFO [StoreOpener-5067d98ca5f8d3760c5d0eb2b8bb06dc-1] regionserver.HStore(310): Store=5067d98ca5f8d3760c5d0eb2b8bb06dc/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-19 20:55:13,515 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:13,515 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc 2023-04-19 20:55:13,516 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc 2023-04-19 20:55:13,518 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-19 20:55:13,518 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 5067d98ca5f8d3760c5d0eb2b8bb06dc 2023-04-19 20:55:13,519 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 5067d98ca5f8d3760c5d0eb2b8bb06dc; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59433864, jitterRate=-0.11436641216278076}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-19 20:55:13,519 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 5067d98ca5f8d3760c5d0eb2b8bb06dc: 2023-04-19 20:55:13,520 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc., pid=42, masterSystemTime=1681937713505 2023-04-19 20:55:13,521 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:13,521 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:13,521 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=5067d98ca5f8d3760c5d0eb2b8bb06dc, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:13,521 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681937713521"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681937713521"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681937713521"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681937713521"}]},"ts":"1681937713521"} 2023-04-19 20:55:13,523 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=42, resume processing ppid=40 2023-04-19 20:55:13,523 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=42, ppid=40, state=SUCCESS; OpenRegionProcedure 5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 in 168 msec 2023-04-19 20:55:13,524 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=40, resume processing ppid=39 2023-04-19 20:55:13,524 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=40, ppid=39, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=5067d98ca5f8d3760c5d0eb2b8bb06dc, REOPEN/MOVE in 484 msec 2023-04-19 20:55:13,525 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=39, resume processing ppid=38 2023-04-19 20:55:13,525 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=39, ppid=38, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 487 msec 2023-04-19 20:55:13,527 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=38, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 901 msec 2023-04-19 20:55:13,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1227): Checking to see if procedure is done pid=38 2023-04-19 20:55:13,528 DEBUG [Listener at localhost/34893] zookeeper.ReadOnlyZKClient(139): Connect 0x2deaf1fa to 127.0.0.1:58736 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-19 20:55:13,533 DEBUG [Listener at localhost/34893] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4c1d19b3, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-19 20:55:13,533 DEBUG [Listener at localhost/34893] zookeeper.ReadOnlyZKClient(139): Connect 0x6e7e18ef to 127.0.0.1:58736 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-19 20:55:13,536 DEBUG [Listener at localhost/34893] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@14e151fb, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-19 20:55:13,537 DEBUG [Listener at localhost/34893] zookeeper.ReadOnlyZKClient(139): Connect 0x5b7d4a51 to 127.0.0.1:58736 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-19 20:55:13,540 DEBUG [Listener at localhost/34893] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@455a88b1, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-19 20:55:13,541 DEBUG [Listener at localhost/34893] zookeeper.ReadOnlyZKClient(139): Connect 0x4335c1f2 to 127.0.0.1:58736 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-19 20:55:13,544 DEBUG [Listener at localhost/34893] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@722b2407, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-19 20:55:13,545 DEBUG [Listener at localhost/34893] zookeeper.ReadOnlyZKClient(139): Connect 0x6bbc9fc8 to 127.0.0.1:58736 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-19 20:55:13,548 DEBUG [Listener at localhost/34893] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@66bf91c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-19 20:55:13,548 DEBUG [Listener at localhost/34893] zookeeper.ReadOnlyZKClient(139): Connect 0x6846dddb to 127.0.0.1:58736 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-19 20:55:13,553 DEBUG [Listener at localhost/34893] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@313c7ad6, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-19 20:55:13,553 DEBUG [Listener at localhost/34893] zookeeper.ReadOnlyZKClient(139): Connect 0x7ea43962 to 127.0.0.1:58736 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-19 20:55:13,557 DEBUG [Listener at localhost/34893] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@20d28b68, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-19 20:55:13,557 DEBUG [Listener at localhost/34893] zookeeper.ReadOnlyZKClient(139): Connect 0x2abd7961 to 127.0.0.1:58736 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-19 20:55:13,568 DEBUG [Listener at localhost/34893] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@750d2a7d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-19 20:55:13,569 DEBUG [Listener at localhost/34893] zookeeper.ReadOnlyZKClient(139): Connect 0x660630e8 to 127.0.0.1:58736 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-19 20:55:13,572 DEBUG [Listener at localhost/34893] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@27313b38, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-19 20:55:13,573 DEBUG [Listener at localhost/34893] zookeeper.ReadOnlyZKClient(139): Connect 0x793239ee to 127.0.0.1:58736 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-19 20:55:13,578 DEBUG [Listener at localhost/34893] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2ee4c750, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-19 20:55:13,588 DEBUG [hconnection-0x5b6956c2-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-19 20:55:13,588 DEBUG [hconnection-0x4f52a090-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-19 20:55:13,588 DEBUG [hconnection-0x41ff350a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-19 20:55:13,589 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:13,589 DEBUG [hconnection-0x30eceebc-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-19 20:55:13,589 DEBUG [hconnection-0x72b51559-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-19 20:55:13,590 DEBUG [hconnection-0x58a30549-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-19 20:55:13,590 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56612, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-19 20:55:13,590 DEBUG [hconnection-0x62b63287-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-19 20:55:13,591 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56642, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-19 20:55:13,591 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56670, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-19 20:55:13,593 DEBUG [hconnection-0x7975222e-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-19 20:55:13,593 DEBUG [hconnection-0x74e456c3-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-19 20:55:13,594 DEBUG [hconnection-0x20da9911-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-19 20:55:13,595 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56610, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-19 20:55:13,597 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56654, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-19 20:55:13,596 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56626, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-19 20:55:13,598 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56676, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-19 20:55:13,598 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56660, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-19 20:55:13,598 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56686, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-19 20:55:13,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:13,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:13,605 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:13,605 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:13,606 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:13,606 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:13,606 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56682, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-19 20:55:13,606 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:13,608 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:13,612 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:13,612 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:13,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:13,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:13,613 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:13,613 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:13,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:13,614 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:13,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:13,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:13,618 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:13,623 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:55:13,623 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:13,624 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:13,624 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:13,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(9158): Flush requested on 5067d98ca5f8d3760c5d0eb2b8bb06dc 2023-04-19 20:55:13,624 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 5067d98ca5f8d3760c5d0eb2b8bb06dc 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-19 20:55:13,624 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:13,626 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5067d98ca5f8d3760c5d0eb2b8bb06dc, store=A 2023-04-19 20:55:13,626 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:55:13,626 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5067d98ca5f8d3760c5d0eb2b8bb06dc, store=B 2023-04-19 20:55:13,626 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:55:13,626 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5067d98ca5f8d3760c5d0eb2b8bb06dc, store=C 2023-04-19 20:55:13,626 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:55:13,635 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:13,635 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:13,635 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:13,635 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. started... 2023-04-19 20:55:13,635 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. as already flushing 2023-04-19 20:55:13,635 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:13,635 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:13,635 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:13,635 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:13,636 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:13,636 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:13,637 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:13,637 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:13,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56654 deadline: 1681937773636, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:13,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56670 deadline: 1681937773636, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:13,637 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:13,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56610 deadline: 1681937773636, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:13,638 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:13,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56626 deadline: 1681937773637, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:13,638 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:13,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56676 deadline: 1681937773637, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:13,639 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:13,639 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:13,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:13,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:13,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:13,639 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:13,639 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:13,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:13,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:13,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:13,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:13,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:13,640 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:13,640 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4ec01d7e[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:13,640 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:13,641 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:13,642 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:13,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:13,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:13,642 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:13,642 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:13,642 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:13,642 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:13,642 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:13,643 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:13,643 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:13,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:13,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:13,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:13,643 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:13,643 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:13,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:13,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:13,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:13,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:13,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:13,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:13,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:13,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:13,646 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:13,646 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:13,646 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:13,646 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:13,646 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:13,646 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:13,648 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:13,648 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:13,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:13,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:13,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:13,648 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:13,648 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:13,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:13,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:13,648 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:13,648 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:13,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:13,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:13,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:13,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:13,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:13,649 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:13,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:13,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:13,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:13,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:13,654 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304198e2c5f7ae35f4aa68fa98debd1d7cca9 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304198e2c5f7ae35f4aa68fa98debd1d7cca9 2023-04-19 20:55:13,657 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:13,657 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:13,657 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:13,657 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:55:13,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:13,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:13,657 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:13,657 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:13,657 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:13,657 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:13,657 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:13,657 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:13,657 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:13,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:13,657 DEBUG [Thread-2269] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:13,657 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:13,657 DEBUG [Thread-2269] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:13,657 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:13,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:13,742 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:13,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56654 deadline: 1681937773738, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:13,742 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:13,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56670 deadline: 1681937773738, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:13,742 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:13,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56610 deadline: 1681937773738, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:13,743 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:13,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56626 deadline: 1681937773738, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:13,743 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:13,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56676 deadline: 1681937773739, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:13,757 DEBUG [Thread-2269] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:13,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:13,761 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:13,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:13,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:13,764 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:13,764 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:13,764 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:13,764 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:13,764 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:13,765 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:13,766 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:13,766 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:13,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:13,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:13,766 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:13,766 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:13,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:13,767 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:13,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:13,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:13,768 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:13,768 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:55:13,768 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:13,768 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:13,768 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:13,768 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:13,772 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:13,772 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:13,772 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:13,772 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. started... 2023-04-19 20:55:13,772 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. as already flushing 2023-04-19 20:55:13,772 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:13,772 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:13,772 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:13,772 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:13,774 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:13,774 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:13,776 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:13,776 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:13,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:13,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:13,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:13,776 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:13,776 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:13,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:13,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:13,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:13,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:13,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:13,777 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:13,777 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3bdb2ed4[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:13,777 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:13,777 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:13,782 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:13,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:13,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:13,782 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:13,782 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:13,782 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:13,782 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:13,782 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:13,784 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:13,784 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:13,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:13,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:13,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:13,784 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:13,784 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:13,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:13,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:13,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:13,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:13,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:13,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:13,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:13,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:13,786 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:13,786 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:13,786 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:13,786 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:13,786 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:13,786 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:13,788 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:13,788 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:13,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:13,788 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:13,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:13,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:13,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:13,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:13,788 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:13,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:13,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:13,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:13,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:13,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:13,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:13,789 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:13,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:13,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:13,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:13,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:13,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:13,792 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:13,792 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:13,792 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:13,792 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:13,792 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:13,792 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:13,792 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:13,793 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:13,792 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:13,793 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:55:13,792 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:13,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:13,793 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:13,793 DEBUG [Thread-2269] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:13,793 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:13,793 DEBUG [Thread-2269] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:13,793 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:13,793 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:13,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:13,893 DEBUG [Thread-2269] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:13,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:13,895 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:13,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:13,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:13,901 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:13,901 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:13,901 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:13,901 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:13,902 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:13,902 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:13,903 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:13,903 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:13,903 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:13,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:13,903 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:13,904 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:13,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:13,904 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:13,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:13,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:13,904 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:13,910 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:55:13,910 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:13,910 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:13,911 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:13,911 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:13,912 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:13,912 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:13,912 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:13,912 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. started... 2023-04-19 20:55:13,912 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. as already flushing 2023-04-19 20:55:13,912 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:13,912 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:13,912 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:13,912 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:13,912 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:13,912 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:13,914 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:13,914 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:13,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:13,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:13,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:13,914 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:13,914 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:13,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:13,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:13,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:13,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:13,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:13,916 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:13,916 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@29c82ec3[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:13,916 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:13,916 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:13,917 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:13,917 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:13,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:13,918 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:13,918 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:13,918 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:13,918 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:13,918 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:13,921 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:13,921 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:13,921 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:13,921 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:13,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:13,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:13,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:13,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:13,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:13,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:13,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:13,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:13,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:13,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:13,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:13,923 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:13,923 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:13,923 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:13,923 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:13,923 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:13,924 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:13,925 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:13,925 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:13,925 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:13,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:13,925 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:13,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:13,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:13,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:13,925 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:13,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:13,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:13,926 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:13,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:13,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:13,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:13,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:13,927 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:13,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:13,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:13,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:13,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:13,932 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:13,932 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:13,932 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:13,932 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:13,932 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:13,932 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:13,932 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:13,932 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:13,932 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:13,932 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:13,932 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:13,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:13,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:13,932 DEBUG [Thread-2269] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:13,932 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:13,932 DEBUG [Thread-2269] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:13,932 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:55:13,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:13,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:13,944 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:13,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56654 deadline: 1681937773943, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:13,945 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:13,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56610 deadline: 1681937773943, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:13,950 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:13,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56670 deadline: 1681937773944, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:13,950 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:13,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56626 deadline: 1681937773944, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:13,950 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:13,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56676 deadline: 1681937773945, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:14,032 DEBUG [Thread-2269] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:14,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:14,035 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:14,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:14,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:14,038 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:14,038 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:14,038 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:14,038 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:14,038 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:14,038 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:14,039 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:14,039 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:14,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:14,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:14,041 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:14,041 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:14,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:14,041 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:14,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:14,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:14,041 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:14,042 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:55:14,042 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:14,042 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:14,042 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:14,042 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:14,042 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:14,042 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:14,042 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:14,042 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. started... 2023-04-19 20:55:14,042 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. as already flushing 2023-04-19 20:55:14,042 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:14,042 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:14,042 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:14,042 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:14,042 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:14,042 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:14,046 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:14,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:14,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:14,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:14,046 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:14,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:14,046 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:14,046 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:14,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:14,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:14,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:14,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:14,048 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:14,048 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@12eaf0c4[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:14,048 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:14,048 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:14,049 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:14,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:14,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:14,049 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:14,049 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:14,049 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:14,049 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:14,049 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:14,051 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:14,051 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:14,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:14,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:14,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:14,051 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:14,051 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:14,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:14,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:14,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:14,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:14,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:14,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:14,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:14,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:14,053 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:14,053 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:14,053 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:14,053 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:14,053 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:14,053 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:14,055 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:14,055 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:14,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:14,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:14,055 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:14,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:14,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:14,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:14,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:14,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:14,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:14,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:14,056 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:14,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:14,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:14,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:14,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:14,057 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:14,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:14,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:14,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:14,061 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=18, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp/A/7a9c45df3a7241a68697ab5f440c3203 2023-04-19 20:55:14,062 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:14,062 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:14,062 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:14,062 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:14,062 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:14,062 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:55:14,062 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:14,062 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:14,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:14,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:14,062 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:14,062 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:14,062 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:14,062 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:14,062 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:14,062 DEBUG [Thread-2269] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:14,062 DEBUG [Thread-2269] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:14,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:14,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:14,079 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-19 20:55:14,163 DEBUG [Thread-2269] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:14,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:14,165 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:14,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:14,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:14,166 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:14,166 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:14,166 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:14,166 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:14,167 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:14,167 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:14,168 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:14,168 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:14,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:14,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:14,168 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:14,169 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:14,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:14,169 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:14,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:14,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:14,169 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:14,169 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:14,170 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:14,170 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:14,170 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:14,170 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:14,170 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:14,170 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:14,170 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. started... 2023-04-19 20:55:14,170 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. as already flushing 2023-04-19 20:55:14,170 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:14,170 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:14,170 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:14,170 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:14,170 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:14,170 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:14,172 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:14,172 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:14,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:14,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:14,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:14,173 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:14,173 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:14,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:14,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:14,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:14,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:14,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:14,174 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:14,174 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c8f340a[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:14,174 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:14,174 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:14,176 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:14,176 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:14,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:14,176 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:14,176 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:14,176 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:14,176 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:14,176 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:14,181 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:14,181 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:14,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:14,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:14,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:14,181 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:14,181 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:14,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:14,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:14,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:14,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:14,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:14,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:14,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:14,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:14,183 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:14,183 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:14,183 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:14,183 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:14,183 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:14,183 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:14,185 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:14,185 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:14,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:14,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:14,185 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:14,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:14,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:14,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:14,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:14,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:14,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:14,186 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:14,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:14,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:14,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:14,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:14,187 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:14,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:14,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:14,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:14,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:14,189 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:14,189 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:14,189 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:14,189 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:14,189 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:14,189 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:14,189 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:14,189 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:14,189 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:14,189 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:14,189 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:55:14,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:14,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:14,189 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:14,190 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:14,190 DEBUG [Thread-2269] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:14,190 DEBUG [Thread-2269] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:14,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:14,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:14,248 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:14,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56654 deadline: 1681937774246, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:14,249 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:14,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56610 deadline: 1681937774246, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:14,253 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:14,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56626 deadline: 1681937774251, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:14,254 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:14,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56670 deadline: 1681937774252, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:14,255 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:14,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56676 deadline: 1681937774253, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:14,290 DEBUG [Thread-2269] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:14,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:14,292 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:14,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:14,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:14,294 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:14,294 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:14,294 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:14,294 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:14,294 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:14,294 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:14,296 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:14,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:14,296 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:14,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:14,297 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:14,297 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:14,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:14,297 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:14,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:14,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:14,297 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:14,298 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:55:14,298 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:14,298 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:14,298 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:14,298 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:14,298 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:14,298 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:14,298 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:14,298 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. started... 2023-04-19 20:55:14,298 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. as already flushing 2023-04-19 20:55:14,298 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:14,298 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:14,298 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:14,298 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:14,298 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:14,298 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:14,300 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:14,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:14,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:14,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:14,300 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:14,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:14,300 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:14,301 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:14,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:14,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:14,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:14,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:14,302 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:14,302 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3e401cc1[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:14,302 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:14,302 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:14,303 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:14,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:14,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:14,303 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:14,303 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:14,303 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:14,303 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:14,303 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:14,306 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:14,306 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:14,306 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:14,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:14,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:14,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:14,306 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:14,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:14,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:14,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:14,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:14,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:14,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:14,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:14,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:14,309 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:14,309 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:14,310 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:14,310 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:14,310 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:14,310 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:14,312 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:14,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:14,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:14,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:14,312 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:14,312 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:14,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:14,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:14,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:14,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:14,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:14,312 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:14,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:14,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:14,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:14,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:14,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:14,314 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:14,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:14,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:14,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:14,316 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:14,316 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:14,316 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:14,316 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:14,316 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:14,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:14,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:14,316 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:14,316 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:14,316 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:14,316 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:14,316 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:14,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:14,317 DEBUG [Thread-2269] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:14,316 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:14,317 DEBUG [Thread-2269] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:14,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:14,317 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:14,417 DEBUG [Thread-2269] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:14,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:14,419 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:14,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:14,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:14,420 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:14,420 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:14,420 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:14,420 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:14,421 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:14,421 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:14,422 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:14,422 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:14,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:14,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:14,422 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:14,422 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:14,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:14,423 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:14,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:14,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:14,423 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:14,423 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:55:14,423 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:14,423 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:14,424 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:14,424 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:14,424 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:14,424 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:14,424 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:14,424 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. started... 2023-04-19 20:55:14,424 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. as already flushing 2023-04-19 20:55:14,424 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:14,424 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:14,424 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:14,424 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:14,424 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:14,424 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:14,426 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:14,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:14,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:14,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:14,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:14,427 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:14,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:14,427 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:14,427 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:14,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:14,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:14,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:14,428 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:14,428 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@561693fc[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:14,428 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:14,428 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:14,430 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:14,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:14,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:14,430 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:14,430 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:14,430 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:14,430 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:14,431 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:14,432 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:14,432 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:14,432 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:14,432 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:14,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:14,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:14,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:14,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:14,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:14,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:14,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:14,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:14,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:14,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:14,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:14,434 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:14,434 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:14,434 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:14,434 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:14,434 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:14,434 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:14,436 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:14,436 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:14,436 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:14,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:14,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:14,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:14,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:14,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:14,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:14,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:14,437 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:14,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:14,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:14,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:14,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:14,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:14,438 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:14,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:14,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:14,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:14,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:14,440 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:14,440 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:14,440 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:55:14,440 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:14,440 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:14,440 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:14,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:14,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:14,440 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:14,441 DEBUG [Thread-2269] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:14,441 DEBUG [Thread-2269] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:14,441 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:14,441 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:14,441 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:14,441 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:14,441 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:14,441 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:14,441 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:14,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:14,472 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=18 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp/B/40a475b1e1ee4d6485edc6590c3764c2 2023-04-19 20:55:14,541 DEBUG [Thread-2269] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:14,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:14,543 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:14,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:14,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:14,544 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:14,544 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:14,544 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:14,544 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:14,545 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:14,545 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:14,547 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:14,547 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:14,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:14,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:14,547 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:14,547 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:14,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:14,548 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:14,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:14,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:14,548 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:14,548 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:55:14,548 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:14,548 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:14,548 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:14,548 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:14,549 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:14,549 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:14,549 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:14,549 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. started... 2023-04-19 20:55:14,549 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. as already flushing 2023-04-19 20:55:14,549 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:14,549 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:14,549 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:14,549 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:14,549 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:14,549 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:14,550 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:14,550 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:14,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:14,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:14,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:14,551 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:14,551 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:14,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:14,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:14,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:14,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:14,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:14,552 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:14,552 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@133f352a[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:14,552 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:14,552 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:14,553 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:14,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:14,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:14,554 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:14,554 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:14,554 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:14,554 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:14,554 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:14,555 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:14,555 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:14,555 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:14,555 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:14,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:14,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:14,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:14,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:14,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:14,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:14,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:14,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:14,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:14,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:14,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:14,557 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:14,557 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:14,558 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:14,558 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:14,558 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:14,558 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:14,560 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:14,560 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:14,560 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:14,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:14,560 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:14,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:14,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:14,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:14,560 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:14,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:14,560 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:14,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:14,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:14,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:14,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:14,563 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:14,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:14,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:14,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:14,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:14,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:14,568 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:14,568 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:14,568 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:14,568 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:14,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:14,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:14,568 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:14,568 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:14,568 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:14,569 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:55:14,568 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:14,569 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:14,569 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:14,569 DEBUG [Thread-2269] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:14,569 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:14,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:14,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:14,569 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:14,569 DEBUG [Thread-2269] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:14,669 DEBUG [Thread-2269] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:14,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:14,671 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:14,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:14,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:14,673 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:14,673 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:14,673 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:14,673 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:14,673 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:14,673 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:14,675 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:14,675 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:14,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:14,675 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:14,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:14,675 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:14,675 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:14,675 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:14,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:14,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:14,676 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:14,676 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:55:14,676 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:14,676 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:14,676 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:14,676 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:14,676 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:14,676 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:14,677 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:14,677 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. started... 2023-04-19 20:55:14,677 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. as already flushing 2023-04-19 20:55:14,677 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:14,677 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:14,677 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:14,677 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:14,677 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:14,677 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:14,678 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:14,678 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:14,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:14,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:14,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:14,679 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:14,679 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:14,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:14,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:14,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:14,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:14,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:14,680 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:14,680 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@36356c23[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:14,680 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:14,680 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:14,682 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:14,682 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:14,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:14,682 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:14,682 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:14,682 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:14,682 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:14,682 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:14,684 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:14,684 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:14,684 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:14,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:14,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:14,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:14,684 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:14,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:14,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:14,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:14,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:14,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:14,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:14,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:14,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:14,686 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:14,686 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:14,686 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:14,686 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:14,687 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:14,687 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:14,688 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:14,688 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:14,688 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:14,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:14,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:14,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:14,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:14,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:14,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:14,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:14,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:14,689 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:14,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:14,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:14,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:14,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:14,690 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:14,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:14,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:14,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:14,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:14,692 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:14,692 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:14,692 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:14,692 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:55:14,692 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:14,692 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:14,692 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:14,692 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:14,692 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:14,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:14,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:14,692 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:14,693 DEBUG [Thread-2269] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:14,693 DEBUG [Thread-2269] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:14,693 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:14,693 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:14,693 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:14,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:14,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:14,756 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:14,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56610 deadline: 1681937774752, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:14,756 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:14,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56654 deadline: 1681937774752, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:14,759 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:14,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56670 deadline: 1681937774756, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:14,763 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:14,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56626 deadline: 1681937774758, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:14,764 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:14,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56676 deadline: 1681937774760, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:14,793 DEBUG [Thread-2269] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:14,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:14,795 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:14,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:14,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:14,796 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:14,796 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:14,796 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:14,796 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:14,796 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:14,796 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:14,799 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:14,799 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:14,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:14,799 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:14,800 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:14,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:14,800 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:14,800 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:14,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:14,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:14,800 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:14,800 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:55:14,800 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:14,800 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:14,801 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:14,801 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:14,801 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:14,801 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:14,801 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:14,801 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. started... 2023-04-19 20:55:14,801 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. as already flushing 2023-04-19 20:55:14,801 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:14,801 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:14,801 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:14,801 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:14,801 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:14,801 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:14,803 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:14,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:14,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:14,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:14,803 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:14,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:14,803 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:14,803 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:14,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:14,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:14,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:14,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:14,804 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:14,804 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@8adc830[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:14,804 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:14,804 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:14,805 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:14,805 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:14,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:14,806 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:14,806 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:14,806 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:14,806 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:14,806 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:14,807 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:14,807 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:14,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:14,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:14,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:14,807 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:14,807 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:14,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:14,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:14,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:14,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:14,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:14,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:14,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:14,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:14,809 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:14,809 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:14,809 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:14,809 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:14,809 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:14,809 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:14,812 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:14,812 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:14,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:14,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:14,812 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:14,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:14,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:14,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:14,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:14,812 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:14,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:14,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:14,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:14,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:14,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:14,813 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:14,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:14,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:14,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:14,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:14,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:14,815 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:14,815 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:14,815 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:14,815 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:14,816 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:14,816 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:14,816 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:14,816 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:14,816 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:14,816 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:14,816 DEBUG [Thread-2269] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:14,816 DEBUG [Thread-2269] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:14,816 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:14,816 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:14,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:14,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:14,816 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:55:14,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:14,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:14,871 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-04-19 20:55:14,884 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=18 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp/C/2c6297cece2044c4bae3dd80658097ed 2023-04-19 20:55:14,888 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp/A/7a9c45df3a7241a68697ab5f440c3203 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/7a9c45df3a7241a68697ab5f440c3203 2023-04-19 20:55:14,892 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/7a9c45df3a7241a68697ab5f440c3203, entries=250, sequenceid=18, filesize=38.7 K 2023-04-19 20:55:14,893 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp/B/40a475b1e1ee4d6485edc6590c3764c2 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/40a475b1e1ee4d6485edc6590c3764c2 2023-04-19 20:55:14,896 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/40a475b1e1ee4d6485edc6590c3764c2, entries=150, sequenceid=18, filesize=11.6 K 2023-04-19 20:55:14,897 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp/C/2c6297cece2044c4bae3dd80658097ed as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/2c6297cece2044c4bae3dd80658097ed 2023-04-19 20:55:14,900 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/2c6297cece2044c4bae3dd80658097ed, entries=150, sequenceid=18, filesize=11.6 K 2023-04-19 20:55:14,901 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 5067d98ca5f8d3760c5d0eb2b8bb06dc in 1277ms, sequenceid=18, compaction requested=false 2023-04-19 20:55:14,902 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 5067d98ca5f8d3760c5d0eb2b8bb06dc: 2023-04-19 20:55:14,916 DEBUG [Thread-2269] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:14,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:14,918 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:14,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:14,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:14,919 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:14,919 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:14,919 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:14,919 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:14,920 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:14,920 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:14,921 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:14,921 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:14,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:14,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:14,922 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:14,922 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:14,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:14,923 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:14,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:14,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:14,924 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:14,924 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:55:14,924 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:14,924 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:14,924 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:14,924 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:14,924 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:14,924 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:14,924 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:14,924 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. started... 2023-04-19 20:55:14,925 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5067d98ca5f8d3760c5d0eb2b8bb06dc 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-19 20:55:14,925 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5067d98ca5f8d3760c5d0eb2b8bb06dc, store=A 2023-04-19 20:55:14,925 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:55:14,925 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5067d98ca5f8d3760c5d0eb2b8bb06dc, store=B 2023-04-19 20:55:14,925 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:55:14,925 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5067d98ca5f8d3760c5d0eb2b8bb06dc, store=C 2023-04-19 20:55:14,925 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:55:14,937 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230419099208ffcc52453da732f4af1e8c9166 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419099208ffcc52453da732f4af1e8c9166 2023-04-19 20:55:15,343 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=36, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp/A/ea3c1d41454940329667338ab43ec20c 2023-04-19 20:55:15,359 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=36 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp/B/714fd4c7d8e243a39c0c8f83118f4ebe 2023-04-19 20:55:15,375 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=36 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp/C/bc27b0c6aec54ca2b7e4ddf1011d33db 2023-04-19 20:55:15,379 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp/A/ea3c1d41454940329667338ab43ec20c as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/ea3c1d41454940329667338ab43ec20c 2023-04-19 20:55:15,382 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/ea3c1d41454940329667338ab43ec20c, entries=150, sequenceid=36, filesize=25.1 K 2023-04-19 20:55:15,383 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp/B/714fd4c7d8e243a39c0c8f83118f4ebe as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/714fd4c7d8e243a39c0c8f83118f4ebe 2023-04-19 20:55:15,387 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/714fd4c7d8e243a39c0c8f83118f4ebe, entries=150, sequenceid=36, filesize=11.6 K 2023-04-19 20:55:15,388 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp/C/bc27b0c6aec54ca2b7e4ddf1011d33db as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/bc27b0c6aec54ca2b7e4ddf1011d33db 2023-04-19 20:55:15,398 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/bc27b0c6aec54ca2b7e4ddf1011d33db, entries=150, sequenceid=36, filesize=11.6 K 2023-04-19 20:55:15,398 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=0 B/0 for 5067d98ca5f8d3760c5d0eb2b8bb06dc in 474ms, sequenceid=36, compaction requested=false 2023-04-19 20:55:15,398 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5067d98ca5f8d3760c5d0eb2b8bb06dc: 2023-04-19 20:55:15,398 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:15,399 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:15,399 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:15,399 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:15,399 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:15,399 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:15,400 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:15,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:15,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:15,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:15,400 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:15,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:15,401 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:15,401 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:15,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:15,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:15,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:15,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:15,402 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:15,402 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a4623aa[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:15,402 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:15,402 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:15,404 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:15,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:15,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:15,404 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:15,404 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:15,404 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:15,405 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:15,405 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:15,406 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:15,406 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:15,406 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:15,406 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:15,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:15,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:15,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:15,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:15,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:15,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:15,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:15,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:15,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:15,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:15,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:15,409 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:15,409 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:15,409 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:15,409 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:15,409 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:15,409 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:15,410 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:15,410 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:15,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:15,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:15,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:15,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:15,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:15,410 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:15,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:15,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:15,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:15,411 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:15,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:15,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:15,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:15,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:15,412 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:15,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:15,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:15,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:15,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:15,421 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:15,421 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:15,421 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:15,421 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:15,421 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:15,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:15,421 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:15,421 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:15,421 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:15,421 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:15,421 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:15,422 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:15,421 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:15,422 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:15,422 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:55:15,422 DEBUG [Thread-2269] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:15,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:15,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:15,422 DEBUG [Thread-2269] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:15,522 DEBUG [Thread-2269] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:15,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:15,525 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:15,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:15,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:15,528 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:15,528 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:15,528 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:15,528 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:15,528 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:15,529 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:15,530 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:15,530 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:15,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:15,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:15,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:15,531 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:15,531 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:15,531 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:15,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:15,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:15,532 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:15,532 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:55:15,532 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:15,532 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:15,532 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:15,532 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:15,532 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:15,532 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:15,532 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:15,532 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. started... 2023-04-19 20:55:15,533 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5067d98ca5f8d3760c5d0eb2b8bb06dc: 2023-04-19 20:55:15,533 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:15,533 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:15,533 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:15,533 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:15,533 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:15,533 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:15,534 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:15,534 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:15,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:15,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:15,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:15,535 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:15,535 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:15,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:15,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:15,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:15,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:15,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:15,536 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:15,536 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@bacdad[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:15,537 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:15,537 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:15,539 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:15,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:15,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:15,539 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:15,539 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:15,539 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:15,539 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:15,539 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:15,540 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:15,540 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:15,540 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:15,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:15,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:15,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:15,540 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:15,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:15,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:15,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:15,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:15,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:15,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:15,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:15,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:15,543 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:15,543 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:15,543 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:15,543 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:15,543 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:15,543 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:15,545 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:15,545 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:15,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:15,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:15,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:15,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:15,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:15,545 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:15,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:15,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:15,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:15,546 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:15,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:15,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:15,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:15,547 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:15,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:15,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:15,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:15,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:15,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:15,549 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:15,550 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:15,550 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:15,550 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:15,550 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:15,550 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:15,550 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:15,550 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:15,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:15,550 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:15,550 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:15,550 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:15,550 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:15,550 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:15,550 DEBUG [Thread-2269] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:15,550 DEBUG [Thread-2269] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:15,550 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:55:15,550 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:15,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:15,650 DEBUG [Thread-2269] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:15,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:15,653 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:15,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:15,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:15,654 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:15,654 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:15,654 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:15,654 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:15,654 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:15,655 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:15,657 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:15,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:15,657 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:15,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:15,657 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:15,657 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:15,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:15,658 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:15,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:15,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:15,658 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:15,658 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:55:15,658 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:15,658 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:15,659 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:15,659 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:15,659 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:15,659 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:15,659 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:15,659 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. started... 2023-04-19 20:55:15,659 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5067d98ca5f8d3760c5d0eb2b8bb06dc: 2023-04-19 20:55:15,659 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:15,659 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:15,659 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:15,659 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:15,660 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:15,660 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:15,661 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:15,662 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:15,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:15,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:15,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:15,662 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:15,662 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:15,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:15,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:15,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:15,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:15,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:15,664 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:15,664 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1df3fded[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:15,664 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:15,664 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:15,665 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:15,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:15,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:15,666 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:15,666 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:15,666 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:15,666 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:15,666 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:15,667 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:15,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:15,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:15,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:15,667 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:15,668 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:15,668 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:15,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:15,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:15,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:15,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:15,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:15,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:15,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:15,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:15,671 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:15,671 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:15,671 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:15,671 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:15,671 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:15,671 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:15,674 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:15,674 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:15,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:15,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:15,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:15,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:15,674 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:15,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:15,675 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:15,675 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:15,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:15,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:15,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:15,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:15,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:15,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:15,676 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:15,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:15,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:15,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:15,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:15,678 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:15,679 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:15,679 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:15,679 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:15,678 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:15,679 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:15,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:15,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:15,679 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:15,679 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:15,679 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:55:15,679 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:15,679 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:15,679 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:15,679 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:15,679 DEBUG [Thread-2269] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:15,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:15,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:15,679 DEBUG [Thread-2269] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:15,779 DEBUG [Thread-2269] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:15,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:15,782 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:15,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:15,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:15,783 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:15,783 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:15,783 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:15,783 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:15,784 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:15,784 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:15,785 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:15,785 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:15,785 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:15,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:15,786 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:15,786 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:15,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:15,786 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:15,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:15,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:15,787 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:15,787 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:55:15,787 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:15,787 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:15,787 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:15,787 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:15,787 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:15,787 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:15,788 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:15,788 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. started... 2023-04-19 20:55:15,788 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5067d98ca5f8d3760c5d0eb2b8bb06dc: 2023-04-19 20:55:15,788 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:15,788 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:15,788 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:15,788 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:15,788 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:15,788 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:15,790 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:15,790 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:15,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:15,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:15,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:15,790 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:15,790 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:15,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:15,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:15,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:15,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:15,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:15,792 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:15,792 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7bf94a2b[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:15,792 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:15,792 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:15,796 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:15,796 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:15,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:15,796 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:15,796 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:15,796 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:15,796 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:15,796 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:15,798 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:15,798 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:15,798 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:15,798 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:15,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:15,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:15,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:15,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:15,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:15,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:15,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:15,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:15,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:15,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:15,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:15,800 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:15,800 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:15,800 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:15,801 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:15,801 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:15,801 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:15,802 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:15,802 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:15,802 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:15,802 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:15,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:15,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:15,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:15,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:15,803 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:15,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:15,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:15,803 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:15,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:15,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:15,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:15,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:15,806 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:15,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:15,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:15,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:15,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:15,808 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:15,808 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:15,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:15,808 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:15,809 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:15,808 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:15,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:15,808 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:15,808 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:15,809 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:15,809 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:15,809 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:15,809 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:15,809 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:15,809 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:15,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:15,809 DEBUG [Thread-2269] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:15,809 DEBUG [Thread-2269] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:15,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(9158): Flush requested on 5067d98ca5f8d3760c5d0eb2b8bb06dc 2023-04-19 20:55:15,873 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 5067d98ca5f8d3760c5d0eb2b8bb06dc 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-19 20:55:15,874 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5067d98ca5f8d3760c5d0eb2b8bb06dc, store=A 2023-04-19 20:55:15,874 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:55:15,874 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5067d98ca5f8d3760c5d0eb2b8bb06dc, store=B 2023-04-19 20:55:15,874 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:55:15,874 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5067d98ca5f8d3760c5d0eb2b8bb06dc, store=C 2023-04-19 20:55:15,874 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:55:15,906 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304192acb5515b7de4bb1a2483d65f7a2e563 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304192acb5515b7de4bb1a2483d65f7a2e563 2023-04-19 20:55:15,909 DEBUG [Thread-2269] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:15,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:15,912 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=49, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp/A/4d46d96b97ff4ec8b010199c78ab2aeb 2023-04-19 20:55:15,912 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:15,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:15,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:15,915 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:15,915 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:15,915 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:15,915 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:15,915 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:15,915 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:15,920 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:15,920 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:15,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:15,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:15,920 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:15,920 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:15,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:15,921 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:15,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:15,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:15,921 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:15,921 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:15,922 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:15,922 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:15,922 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:15,922 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:15,922 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:15,922 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:15,922 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. started... 2023-04-19 20:55:15,922 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. as already flushing 2023-04-19 20:55:15,922 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:15,922 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:15,922 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:15,922 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:15,922 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:15,922 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:15,924 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:15,924 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:15,924 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:15,924 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:15,925 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1681937715903, max timestamp=1681937715922Num uniques 859; 2023-04-19 20:55:15,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:15,925 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1681937715903, max timestamp=1681937715922Num uniques 859; 2023-04-19 20:55:15,925 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1681937715903, max timestamp=1681937715922Num uniques 859; 2023-04-19 20:55:15,925 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-19 20:55:15,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:15,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:15,925 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-19 20:55:15,925 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-19 20:55:15,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:15,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:15,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:15,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:15,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:15,927 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:15,927 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@20796d8d[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:15,927 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:15,927 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:15,928 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:15,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:15,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:15,928 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:15,928 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:15,928 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:15,928 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:15,928 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:15,931 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:15,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:15,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:15,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:15,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:15,932 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:15,932 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:15,932 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:15,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:15,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:15,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:15,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:15,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:15,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:15,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:15,934 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:15,934 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:15,934 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:15,934 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:15,934 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:15,934 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:15,935 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:15,935 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:15,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:15,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:15,935 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:15,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:15,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:15,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:15,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:15,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:15,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:15,938 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:15,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:15,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:15,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:15,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:15,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:15,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:15,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:15,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:15,941 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:15,943 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:15,943 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:15,943 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:15,943 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:15,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:15,943 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:15,943 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:15,943 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:15,943 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:15,943 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:15,943 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:15,943 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:15,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:15,943 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:15,943 DEBUG [Thread-2269] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:15,944 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:15,944 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:15,944 DEBUG [Thread-2269] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:15,953 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:15,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56626 deadline: 1681937775943, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:15,954 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:15,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56654 deadline: 1681937775944, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:15,955 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:15,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56670 deadline: 1681937775946, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:15,958 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:15,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56610 deadline: 1681937775953, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:15,961 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:15,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56676 deadline: 1681937775954, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:16,044 DEBUG [Thread-2269] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:16,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:16,046 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:16,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:16,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:16,047 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:16,047 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:16,047 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:16,047 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:16,048 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:16,048 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:16,049 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:16,049 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:16,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:16,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:16,049 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:16,049 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:16,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:16,050 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:16,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:16,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:16,050 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:16,050 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:55:16,050 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:16,050 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:16,051 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:16,051 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:16,051 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:16,051 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:16,051 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:16,051 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. started... 2023-04-19 20:55:16,051 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. as already flushing 2023-04-19 20:55:16,051 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:16,051 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:16,051 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:16,051 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:16,051 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:16,051 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:16,053 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:16,053 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:16,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:16,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:16,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:16,054 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:16,054 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:16,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:16,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:16,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:16,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:16,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:16,055 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:16,055 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@76f80b8a[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:16,055 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:16,055 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:16,056 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:16,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:16,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:16,057 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:16,057 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:16,060 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:16,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56626 deadline: 1681937776054, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:16,060 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:16,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56654 deadline: 1681937776054, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:16,060 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:16,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56670 deadline: 1681937776056, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:16,063 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:16,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56610 deadline: 1681937776059, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:16,064 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:16,064 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:16,064 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:16,065 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:16,065 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:16,065 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:16,065 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:16,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:16,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:16,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:16,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:16,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:16,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:16,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:16,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:16,067 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:16,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56676 deadline: 1681937776061, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:16,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:16,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:16,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:16,068 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:16,068 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:16,068 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:16,068 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:16,068 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:16,068 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:16,069 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:16,069 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:16,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:16,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:16,069 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:16,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:16,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:16,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:16,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:16,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:16,070 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:16,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:16,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:16,071 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:16,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:16,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:16,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:16,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:16,074 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:16,075 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:16,075 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:16,075 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:16,075 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:16,075 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:16,075 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:16,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:16,075 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:16,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:16,075 DEBUG [Thread-2269] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:16,075 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:16,075 DEBUG [Thread-2269] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:16,075 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:16,075 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:16,075 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:16,075 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:16,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:16,075 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:16,076 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:55:16,175 DEBUG [Thread-2269] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:16,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:16,177 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:16,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:16,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:16,179 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:16,179 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:16,179 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:16,179 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:16,179 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:16,179 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:16,181 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:16,181 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:16,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:16,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:16,181 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:16,181 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:16,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:16,181 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:16,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:16,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:16,182 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:16,182 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:55:16,182 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:16,182 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:16,182 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:16,182 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:16,182 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:16,182 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:16,182 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:16,182 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. started... 2023-04-19 20:55:16,182 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. as already flushing 2023-04-19 20:55:16,183 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:16,183 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:16,183 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:16,183 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:16,183 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:16,183 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:16,184 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:16,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:16,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:16,184 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:16,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:16,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:16,185 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:16,185 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:16,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:16,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:16,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:16,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:16,186 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:16,186 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@74e69c2d[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:16,186 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:16,186 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:16,187 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:16,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:16,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:16,188 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:16,188 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:16,188 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:16,188 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:16,188 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:16,190 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:16,190 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:16,190 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:16,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:16,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:16,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:16,190 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:16,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:16,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:16,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:16,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:16,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:16,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:16,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:16,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:16,193 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:16,193 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:16,193 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:16,193 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:16,193 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:16,193 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:16,195 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:16,195 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:16,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:16,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:16,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:16,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:16,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:16,195 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:16,195 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:16,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:16,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:16,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:16,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:16,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:16,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:16,196 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:16,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:16,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:16,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:16,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:16,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:16,201 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:16,201 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:16,201 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:16,201 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:16,201 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:16,201 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:16,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:16,201 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:16,201 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:16,201 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:16,201 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:16,201 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:16,201 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:16,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:16,201 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:55:16,201 DEBUG [Thread-2269] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:16,201 DEBUG [Thread-2269] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:16,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:16,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:16,265 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:16,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56626 deadline: 1681937776262, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:16,265 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:16,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56654 deadline: 1681937776262, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:16,269 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:16,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56610 deadline: 1681937776265, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:16,270 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:16,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56670 deadline: 1681937776265, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:16,271 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:16,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56676 deadline: 1681937776268, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:16,301 DEBUG [Thread-2269] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:16,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:16,303 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:16,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:16,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:16,305 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:16,305 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:16,305 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:16,305 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:16,306 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:16,306 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:16,307 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:16,307 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:16,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:16,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:16,307 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:16,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:16,307 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:16,307 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:16,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:16,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:16,308 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:16,308 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:55:16,308 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:16,308 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:16,308 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:16,308 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:16,309 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:16,309 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:16,309 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:16,309 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. started... 2023-04-19 20:55:16,309 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. as already flushing 2023-04-19 20:55:16,309 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:16,309 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:16,309 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:16,309 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:16,309 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:16,309 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:16,313 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:16,314 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:16,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:16,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:16,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:16,314 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:16,314 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:16,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:16,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:16,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:16,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:16,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:16,315 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:16,315 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@45ff3888[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:16,315 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:16,315 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:16,316 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:16,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:16,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:16,317 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:16,317 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:16,317 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:16,317 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:16,317 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:16,318 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:16,318 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:16,318 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:16,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:16,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:16,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:16,318 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:16,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:16,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:16,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:16,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:16,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:16,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:16,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:16,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:16,321 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:16,321 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:16,321 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:16,321 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:16,321 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:16,321 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:16,322 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=49 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp/B/c94dccc5398c482ca67248b6814adb8e 2023-04-19 20:55:16,325 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:16,325 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:16,325 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:16,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:16,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:16,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:16,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:16,325 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:16,326 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:16,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:16,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:16,326 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:16,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:16,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:16,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:16,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:16,327 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:16,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:16,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:16,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:16,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:16,331 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:16,331 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:16,331 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:16,331 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:16,331 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:16,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:16,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:16,331 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:16,331 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:16,331 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:16,331 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:16,331 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:16,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:16,331 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:16,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:16,331 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:16,331 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:55:16,331 DEBUG [Thread-2269] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:16,331 DEBUG [Thread-2269] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:16,332 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=49 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp/C/8e7ba01953844fb8951f5c20644aee91 2023-04-19 20:55:16,335 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp/A/4d46d96b97ff4ec8b010199c78ab2aeb as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/4d46d96b97ff4ec8b010199c78ab2aeb 2023-04-19 20:55:16,340 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/4d46d96b97ff4ec8b010199c78ab2aeb, entries=150, sequenceid=49, filesize=25.1 K 2023-04-19 20:55:16,340 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp/B/c94dccc5398c482ca67248b6814adb8e as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/c94dccc5398c482ca67248b6814adb8e 2023-04-19 20:55:16,344 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/c94dccc5398c482ca67248b6814adb8e, entries=100, sequenceid=49, filesize=9.3 K 2023-04-19 20:55:16,345 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp/C/8e7ba01953844fb8951f5c20644aee91 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/8e7ba01953844fb8951f5c20644aee91 2023-04-19 20:55:16,348 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/8e7ba01953844fb8951f5c20644aee91, entries=100, sequenceid=49, filesize=9.3 K 2023-04-19 20:55:16,348 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=140.89 KB/144270 for 5067d98ca5f8d3760c5d0eb2b8bb06dc in 475ms, sequenceid=49, compaction requested=true 2023-04-19 20:55:16,348 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 5067d98ca5f8d3760c5d0eb2b8bb06dc: 2023-04-19 20:55:16,348 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 20:55:16,348 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-19 20:55:16,348 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-19 20:55:16,348 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 20:55:16,349 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 20:55:16,349 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 91118 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-19 20:55:16,350 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1912): 5067d98ca5f8d3760c5d0eb2b8bb06dc/A is initiating minor compaction (all files) 2023-04-19 20:55:16,350 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 33314 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-19 20:55:16,350 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 5067d98ca5f8d3760c5d0eb2b8bb06dc/A in TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:16,350 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1912): 5067d98ca5f8d3760c5d0eb2b8bb06dc/B is initiating minor compaction (all files) 2023-04-19 20:55:16,350 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/7a9c45df3a7241a68697ab5f440c3203, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/ea3c1d41454940329667338ab43ec20c, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/4d46d96b97ff4ec8b010199c78ab2aeb] into tmpdir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp, totalSize=89.0 K 2023-04-19 20:55:16,350 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5067d98ca5f8d3760c5d0eb2b8bb06dc/B in TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:16,350 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/40a475b1e1ee4d6485edc6590c3764c2, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/714fd4c7d8e243a39c0c8f83118f4ebe, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/c94dccc5398c482ca67248b6814adb8e] into tmpdir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp, totalSize=32.5 K 2023-04-19 20:55:16,350 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 7a9c45df3a7241a68697ab5f440c3203, keycount=250, bloomtype=ROW, size=38.7 K, encoding=NONE, compression=NONE, seqNum=18, earliestPutTs=1681937713614 2023-04-19 20:55:16,350 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 40a475b1e1ee4d6485edc6590c3764c2, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=18, earliestPutTs=1681937713614 2023-04-19 20:55:16,350 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting ea3c1d41454940329667338ab43ec20c, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=36, earliestPutTs=1681937713634 2023-04-19 20:55:16,350 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 714fd4c7d8e243a39c0c8f83118f4ebe, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=36, earliestPutTs=1681937713634 2023-04-19 20:55:16,351 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 4d46d96b97ff4ec8b010199c78ab2aeb, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=49, earliestPutTs=1681937715832 2023-04-19 20:55:16,351 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting c94dccc5398c482ca67248b6814adb8e, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=49, earliestPutTs=1681937715832 2023-04-19 20:55:16,357 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] throttle.PressureAwareThroughputController(145): 5067d98ca5f8d3760c5d0eb2b8bb06dc#B#compaction#306 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 57.69 MB/second 2023-04-19 20:55:16,358 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] throttle.PressureAwareThroughputController(145): 5067d98ca5f8d3760c5d0eb2b8bb06dc#A#compaction#305 average throughput is 6.57 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 57.69 MB/second 2023-04-19 20:55:16,373 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp/B/f9c06ee3de7043fb945995fb46b3a40f as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/f9c06ee3de7043fb945995fb46b3a40f 2023-04-19 20:55:16,377 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5067d98ca5f8d3760c5d0eb2b8bb06dc/B of 5067d98ca5f8d3760c5d0eb2b8bb06dc into f9c06ee3de7043fb945995fb46b3a40f(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 20:55:16,377 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5067d98ca5f8d3760c5d0eb2b8bb06dc: 2023-04-19 20:55:16,377 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc., storeName=5067d98ca5f8d3760c5d0eb2b8bb06dc/B, priority=13, startTime=1681937716348; duration=0sec 2023-04-19 20:55:16,377 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 20:55:16,377 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-19 20:55:16,378 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 33314 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-19 20:55:16,378 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1912): 5067d98ca5f8d3760c5d0eb2b8bb06dc/C is initiating minor compaction (all files) 2023-04-19 20:55:16,378 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5067d98ca5f8d3760c5d0eb2b8bb06dc/C in TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:16,378 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/2c6297cece2044c4bae3dd80658097ed, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/bc27b0c6aec54ca2b7e4ddf1011d33db, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/8e7ba01953844fb8951f5c20644aee91] into tmpdir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp, totalSize=32.5 K 2023-04-19 20:55:16,379 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 2c6297cece2044c4bae3dd80658097ed, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=18, earliestPutTs=1681937713614 2023-04-19 20:55:16,379 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting bc27b0c6aec54ca2b7e4ddf1011d33db, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=36, earliestPutTs=1681937713634 2023-04-19 20:55:16,379 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 8e7ba01953844fb8951f5c20644aee91, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=49, earliestPutTs=1681937715832 2023-04-19 20:55:16,384 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] throttle.PressureAwareThroughputController(145): 5067d98ca5f8d3760c5d0eb2b8bb06dc#C#compaction#307 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 57.69 MB/second 2023-04-19 20:55:16,432 DEBUG [Thread-2269] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:16,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:16,434 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:16,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:16,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:16,435 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:16,435 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:16,435 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:16,435 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:16,435 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:16,435 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:16,437 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:16,437 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:16,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:16,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:16,437 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:16,437 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:16,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:16,438 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:16,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:16,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:16,438 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:16,438 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:55:16,438 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:16,438 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:16,438 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:16,438 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:16,439 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:16,439 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:16,439 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:16,439 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. started... 2023-04-19 20:55:16,439 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5067d98ca5f8d3760c5d0eb2b8bb06dc 3/3 column families, dataSize=140.89 KB heapSize=390.84 KB 2023-04-19 20:55:16,439 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5067d98ca5f8d3760c5d0eb2b8bb06dc, store=A 2023-04-19 20:55:16,439 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 20:55:16,439 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5067d98ca5f8d3760c5d0eb2b8bb06dc, store=B 2023-04-19 20:55:16,439 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 20:55:16,439 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5067d98ca5f8d3760c5d0eb2b8bb06dc, store=C 2023-04-19 20:55:16,439 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 20:55:16,463 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230419832223214dac47ebb7e4b6713b599801 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419832223214dac47ebb7e4b6713b599801 2023-04-19 20:55:16,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(9158): Flush requested on 5067d98ca5f8d3760c5d0eb2b8bb06dc 2023-04-19 20:55:16,569 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. as already flushing 2023-04-19 20:55:16,583 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:16,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56626 deadline: 1681937776577, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:16,584 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:16,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56610 deadline: 1681937776578, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:16,584 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:16,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56654 deadline: 1681937776578, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:16,584 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:16,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56670 deadline: 1681937776578, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:16,586 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:16,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56676 deadline: 1681937776579, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:16,687 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:16,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56626 deadline: 1681937776684, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:16,687 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:16,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56610 deadline: 1681937776685, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:16,687 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:16,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56654 deadline: 1681937776685, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:16,688 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:16,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56670 deadline: 1681937776685, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:16,691 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:16,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56676 deadline: 1681937776687, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:16,793 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp/C/6d9c33d398a24bbcb7c55c1ee71e29ef as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/6d9c33d398a24bbcb7c55c1ee71e29ef 2023-04-19 20:55:16,805 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5067d98ca5f8d3760c5d0eb2b8bb06dc/C of 5067d98ca5f8d3760c5d0eb2b8bb06dc into 6d9c33d398a24bbcb7c55c1ee71e29ef(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 20:55:16,805 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5067d98ca5f8d3760c5d0eb2b8bb06dc: 2023-04-19 20:55:16,805 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc., storeName=5067d98ca5f8d3760c5d0eb2b8bb06dc/C, priority=13, startTime=1681937716349; duration=0sec 2023-04-19 20:55:16,805 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 20:55:16,869 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=85, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp/A/c062767b9d634d2fbb9737f52c20e15a 2023-04-19 20:55:16,894 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:16,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56626 deadline: 1681937776888, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:16,894 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:16,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56670 deadline: 1681937776888, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:16,894 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:16,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56610 deadline: 1681937776889, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:16,895 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:16,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56654 deadline: 1681937776889, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:16,897 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:16,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56676 deadline: 1681937776893, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:17,201 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:17,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56610 deadline: 1681937777195, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:17,201 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:17,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56654 deadline: 1681937777195, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:17,201 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:17,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56626 deadline: 1681937777196, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:17,201 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:17,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56670 deadline: 1681937777197, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:17,204 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:17,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56676 deadline: 1681937777199, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:17,279 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=85 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp/B/0de8ee6282ed47b4b49617d785bca139 2023-04-19 20:55:17,585 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp/A/ad1315a26f8e4150b205bc0be6b2463a as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/ad1315a26f8e4150b205bc0be6b2463a 2023-04-19 20:55:17,592 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5067d98ca5f8d3760c5d0eb2b8bb06dc/A of 5067d98ca5f8d3760c5d0eb2b8bb06dc into ad1315a26f8e4150b205bc0be6b2463a(size=25.2 K), total size for store is 25.2 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-19 20:55:17,592 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5067d98ca5f8d3760c5d0eb2b8bb06dc: 2023-04-19 20:55:17,592 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc., storeName=5067d98ca5f8d3760c5d0eb2b8bb06dc/A, priority=13, startTime=1681937716348; duration=1sec 2023-04-19 20:55:17,592 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 20:55:17,695 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=85 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp/C/977c283471774b328ccb52d7451ffa12 2023-04-19 20:55:17,699 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp/A/c062767b9d634d2fbb9737f52c20e15a as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/c062767b9d634d2fbb9737f52c20e15a 2023-04-19 20:55:17,704 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/c062767b9d634d2fbb9737f52c20e15a, entries=150, sequenceid=85, filesize=25.1 K 2023-04-19 20:55:17,705 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp/B/0de8ee6282ed47b4b49617d785bca139 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/0de8ee6282ed47b4b49617d785bca139 2023-04-19 20:55:17,708 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:17,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56670 deadline: 1681937777702, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:17,709 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/0de8ee6282ed47b4b49617d785bca139, entries=150, sequenceid=85, filesize=11.6 K 2023-04-19 20:55:17,709 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp/C/977c283471774b328ccb52d7451ffa12 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/977c283471774b328ccb52d7451ffa12 2023-04-19 20:55:17,711 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:17,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56610 deadline: 1681937777704, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:17,712 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:17,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56626 deadline: 1681937777704, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:17,712 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:17,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56654 deadline: 1681937777706, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:17,712 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:17,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56676 deadline: 1681937777707, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:17,714 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/977c283471774b328ccb52d7451ffa12, entries=150, sequenceid=85, filesize=11.6 K 2023-04-19 20:55:17,715 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~390.80 KB/400176, currentSize=46.96 KB/48090 for 5067d98ca5f8d3760c5d0eb2b8bb06dc in 1276ms, sequenceid=85, compaction requested=false 2023-04-19 20:55:17,715 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5067d98ca5f8d3760c5d0eb2b8bb06dc: 2023-04-19 20:55:17,715 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:17,715 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:17,715 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:17,715 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:17,715 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:17,715 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:17,716 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:17,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:17,717 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:17,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:17,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:17,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:17,717 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:17,717 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:17,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:17,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:17,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:17,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:17,718 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:17,718 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2e8120bf[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:17,718 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:17,718 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:17,720 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:17,721 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:17,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:17,721 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:17,721 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:17,721 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:17,721 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:17,721 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:17,722 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:17,722 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:17,722 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:17,723 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:17,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:17,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:17,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:17,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:17,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:17,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:17,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:17,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:17,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:17,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:17,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:17,727 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:17,727 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:17,727 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:17,727 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:17,727 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:17,727 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:17,728 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:17,728 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:17,728 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:17,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:17,729 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:17,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:17,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:17,729 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:17,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:17,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:17,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:17,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:17,730 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:17,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:17,731 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:17,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:17,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:17,734 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:17,734 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:17,734 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:17,734 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:17,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:17,734 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:17,734 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:17,734 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:17,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:17,734 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:17,734 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:17,734 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:17,734 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:17,734 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:17,734 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:17,734 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:17,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:17,734 DEBUG [Thread-2269] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:17,735 DEBUG [Thread-2269] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:17,735 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:55:17,835 DEBUG [Thread-2269] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:17,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:17,837 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:17,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:17,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:17,838 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:17,838 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:17,838 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:17,839 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:17,839 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:17,839 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:17,842 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:17,842 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:17,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:17,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:17,842 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:17,842 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:17,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:17,842 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:17,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:17,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:17,843 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:17,843 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:55:17,843 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:17,843 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:17,843 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:17,843 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:17,844 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:17,844 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:17,844 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:17,844 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. started... 2023-04-19 20:55:17,844 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5067d98ca5f8d3760c5d0eb2b8bb06dc 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-19 20:55:17,844 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5067d98ca5f8d3760c5d0eb2b8bb06dc, store=A 2023-04-19 20:55:17,844 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:55:17,844 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5067d98ca5f8d3760c5d0eb2b8bb06dc, store=B 2023-04-19 20:55:17,844 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:55:17,844 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5067d98ca5f8d3760c5d0eb2b8bb06dc, store=C 2023-04-19 20:55:17,844 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:55:17,864 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230419d20601e9137540faaf4d3f389963337f to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419d20601e9137540faaf4d3f389963337f 2023-04-19 20:55:18,280 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=97, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp/A/c8a8147791b546ac8b0cf5eb79a7cc66 2023-04-19 20:55:18,297 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=97 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp/B/2c76ce6f46344485814067d8447f3a66 2023-04-19 20:55:18,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(9158): Flush requested on 5067d98ca5f8d3760c5d0eb2b8bb06dc 2023-04-19 20:55:18,711 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. as already flushing 2023-04-19 20:55:18,712 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=97 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp/C/adaf97591f734ec69d8eb866392635b5 2023-04-19 20:55:18,716 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp/A/c8a8147791b546ac8b0cf5eb79a7cc66 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/c8a8147791b546ac8b0cf5eb79a7cc66 2023-04-19 20:55:18,719 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/c8a8147791b546ac8b0cf5eb79a7cc66, entries=150, sequenceid=97, filesize=25.1 K 2023-04-19 20:55:18,720 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp/B/2c76ce6f46344485814067d8447f3a66 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/2c76ce6f46344485814067d8447f3a66 2023-04-19 20:55:18,729 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/2c76ce6f46344485814067d8447f3a66, entries=150, sequenceid=97, filesize=11.6 K 2023-04-19 20:55:18,730 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp/C/adaf97591f734ec69d8eb866392635b5 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/adaf97591f734ec69d8eb866392635b5 2023-04-19 20:55:18,734 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/adaf97591f734ec69d8eb866392635b5, entries=150, sequenceid=97, filesize=11.6 K 2023-04-19 20:55:18,734 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=40.25 KB/41220 for 5067d98ca5f8d3760c5d0eb2b8bb06dc in 890ms, sequenceid=97, compaction requested=true 2023-04-19 20:55:18,735 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5067d98ca5f8d3760c5d0eb2b8bb06dc: 2023-04-19 20:55:18,735 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:18,735 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:18,735 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:18,735 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:18,738 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:18,738 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:18,740 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:18,740 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:18,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:18,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:18,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:18,741 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:18,741 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:18,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:18,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:18,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:18,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:18,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:18,742 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:18,742 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1813dabe[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:18,743 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:18,743 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:18,744 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:18,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:18,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:18,744 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:18,744 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:18,744 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:18,744 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:18,744 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:18,745 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:18,745 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:18,745 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:18,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:18,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:18,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:18,745 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:18,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:18,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:18,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:18,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:18,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:18,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:18,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:18,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:18,748 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:18,748 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:18,748 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:18,748 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:18,748 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:18,748 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:18,749 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:18,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:18,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:18,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:18,749 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:18,750 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:18,750 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:18,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:18,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:18,750 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:18,750 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:18,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:18,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:18,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:18,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:18,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:18,751 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:18,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:18,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:18,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:18,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:18,754 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:18,754 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:18,754 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:18,754 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:18,754 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:55:18,754 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:18,754 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:18,754 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:18,754 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:18,755 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:18,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:18,755 DEBUG [Thread-2269] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:18,755 DEBUG [Thread-2269] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:18,755 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:18,755 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:18,755 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:18,755 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:18,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(9158): Flush requested on 5067d98ca5f8d3760c5d0eb2b8bb06dc 2023-04-19 20:55:18,756 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 5067d98ca5f8d3760c5d0eb2b8bb06dc 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-19 20:55:18,756 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:18,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:18,757 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5067d98ca5f8d3760c5d0eb2b8bb06dc, store=A 2023-04-19 20:55:18,757 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:55:18,757 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5067d98ca5f8d3760c5d0eb2b8bb06dc, store=B 2023-04-19 20:55:18,757 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:55:18,757 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5067d98ca5f8d3760c5d0eb2b8bb06dc, store=C 2023-04-19 20:55:18,757 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:55:18,768 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041998988b2aed9549308ee8974217c9f865 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041998988b2aed9549308ee8974217c9f865 2023-04-19 20:55:18,773 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=108, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp/A/f354c78a24c445fd9b41bc8987f232d8 2023-04-19 20:55:18,800 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1681937718784, max timestamp=1681937718798Num uniques 859; 2023-04-19 20:55:18,800 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-04-19 20:55:18,801 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681937718784, max timestamp=1681937718798Num uniques 859; 2023-04-19 20:55:18,801 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-19 20:55:18,801 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681937718784, max timestamp=1681937718798Num uniques 859; 2023-04-19 20:55:18,801 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-19 20:55:18,829 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:18,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56626 deadline: 1681937778820, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:18,833 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:18,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56610 deadline: 1681937778829, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:18,837 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:18,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56676 deadline: 1681937778829, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:18,838 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:18,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56654 deadline: 1681937778829, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:18,838 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:18,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56670 deadline: 1681937778830, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:18,855 DEBUG [Thread-2269] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:18,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:18,857 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:18,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:18,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:18,859 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:18,859 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:18,859 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:18,859 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:18,859 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:18,859 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:18,860 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:18,860 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:18,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:18,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:18,861 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:18,861 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:18,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:18,861 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:18,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:18,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:18,862 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:18,862 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:55:18,862 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:18,862 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:18,862 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:18,862 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:18,863 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:18,863 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:18,863 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:18,863 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. started... 2023-04-19 20:55:18,863 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. as already flushing 2023-04-19 20:55:18,863 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:18,863 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:18,864 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:18,864 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:18,864 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:18,864 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:18,865 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:18,866 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:18,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:18,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:18,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:18,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:18,867 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:18,867 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:18,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:18,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:18,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:18,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:18,868 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:18,868 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:18,868 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@58d42a65[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:18,869 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:18,870 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:18,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:18,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:18,870 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:18,870 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:18,870 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:18,870 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:18,870 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:18,875 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:18,875 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:18,875 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:18,876 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:18,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:18,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:18,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:18,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:18,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:18,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:18,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:18,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:18,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:18,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:18,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:18,879 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:18,879 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:18,879 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:18,879 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:18,879 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:18,879 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:18,881 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:18,881 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:18,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:18,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:18,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:18,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:18,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:18,881 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:18,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:18,882 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:18,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:18,882 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:18,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:18,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:18,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:18,883 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:18,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:18,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:18,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:18,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:18,896 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:18,896 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:18,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:18,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:18,896 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:18,896 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:18,896 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:18,896 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:18,896 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:18,896 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:18,897 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:18,897 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:18,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:18,896 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:18,897 DEBUG [Thread-2269] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:18,897 DEBUG [Thread-2269] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:18,897 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:18,897 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:18,897 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:55:18,936 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:18,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56626 deadline: 1681937778930, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:18,944 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:18,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56610 deadline: 1681937778934, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:18,945 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:18,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56676 deadline: 1681937778938, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:18,945 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:18,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56670 deadline: 1681937778940, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:18,945 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:18,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56654 deadline: 1681937778940, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:18,997 DEBUG [Thread-2269] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:18,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:18,999 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:19,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:19,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:19,000 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:19,000 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:19,000 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:19,000 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:19,001 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:19,001 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:19,002 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:19,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:19,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:19,002 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:19,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:19,003 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:19,003 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:19,003 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:19,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:19,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:19,003 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:19,004 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:19,004 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:19,004 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:19,004 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:19,004 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:19,004 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:19,004 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:19,004 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. started... 2023-04-19 20:55:19,004 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. as already flushing 2023-04-19 20:55:19,004 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:19,004 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:19,004 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:19,004 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:19,004 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:19,004 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:19,007 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:19,007 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:19,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:19,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:19,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:19,007 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:19,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:19,007 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:19,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:19,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:19,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:19,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:19,008 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:19,008 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6453728f[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:19,008 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:19,008 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:19,010 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:19,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:19,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:19,010 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:19,010 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:19,010 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:19,010 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-19 20:55:19,012 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:19,012 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:19,016 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:19,017 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:19,017 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:19,017 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:19,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:19,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:19,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:19,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:19,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:19,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:19,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:19,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:19,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:19,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:19,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:19,019 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:19,020 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:19,020 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:19,020 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:19,020 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:19,020 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:19,021 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:19,021 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:19,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:19,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:19,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:19,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:19,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:19,021 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:19,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:19,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:19,022 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:19,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:19,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:19,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:19,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:19,027 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:19,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:19,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:19,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:19,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:19,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:19,034 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:19,035 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:19,035 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:19,035 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:19,035 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:19,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:19,035 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:19,035 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:19,035 DEBUG [Thread-2269] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:19,035 DEBUG [Thread-2269] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:19,035 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:19,035 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:19,036 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:19,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:19,036 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:19,036 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:19,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:19,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:19,037 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:55:19,136 DEBUG [Thread-2269] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:19,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:19,140 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:19,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:19,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:19,141 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:19,141 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:19,141 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:19,141 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:19,141 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:19,141 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:19,143 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:19,143 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:19,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:19,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:19,143 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:19,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56626 deadline: 1681937779138, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:19,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:19,144 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:19,144 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:19,144 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:19,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:19,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:19,145 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:19,146 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:55:19,146 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:19,146 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:19,146 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:19,146 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:19,146 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:19,146 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:19,146 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:19,146 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. started... 2023-04-19 20:55:19,146 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. as already flushing 2023-04-19 20:55:19,146 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:19,147 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:19,147 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:19,147 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:19,147 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:19,147 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:19,149 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:19,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:19,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:19,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:19,149 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:19,149 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:19,149 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:19,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:19,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:19,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:19,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:19,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:19,151 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:19,151 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@43f02262[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:19,151 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:19,151 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:19,152 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:19,152 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:19,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:19,153 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:19,153 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:19,153 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:19,153 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:19,153 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:19,155 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:19,155 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:19,155 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:19,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:19,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:19,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:19,155 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:19,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:19,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:19,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:19,156 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:19,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56610 deadline: 1681937779146, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:19,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:19,156 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:19,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56654 deadline: 1681937779146, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:19,157 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:19,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:19,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56670 deadline: 1681937779146, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:19,157 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:19,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56676 deadline: 1681937779147, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:19,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:19,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:19,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:19,158 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:19,158 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:19,158 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:19,158 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:19,158 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:19,158 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:19,159 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:19,159 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:19,159 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:19,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:19,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:19,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:19,159 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:19,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:19,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:19,160 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:19,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:19,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:19,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:19,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:19,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:19,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:19,161 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:19,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:19,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:19,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:19,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:19,163 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:19,163 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:19,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:19,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:19,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:19,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:19,164 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:19,164 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:19,164 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:19,164 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:19,164 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:19,164 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:19,164 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:19,164 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:19,164 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:19,164 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:19,164 DEBUG [Thread-2269] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:19,164 DEBUG [Thread-2269] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:19,202 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=108 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp/B/bb55aa0fba75465fb625cbe11c6eadee 2023-04-19 20:55:19,211 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=108 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp/C/b0b7f37b35aa4eb0847fa997dd7e3d43 2023-04-19 20:55:19,214 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp/A/f354c78a24c445fd9b41bc8987f232d8 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/f354c78a24c445fd9b41bc8987f232d8 2023-04-19 20:55:19,217 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/f354c78a24c445fd9b41bc8987f232d8, entries=100, sequenceid=108, filesize=18.4 K 2023-04-19 20:55:19,218 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp/B/bb55aa0fba75465fb625cbe11c6eadee as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/bb55aa0fba75465fb625cbe11c6eadee 2023-04-19 20:55:19,221 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/bb55aa0fba75465fb625cbe11c6eadee, entries=100, sequenceid=108, filesize=9.3 K 2023-04-19 20:55:19,221 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp/C/b0b7f37b35aa4eb0847fa997dd7e3d43 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/b0b7f37b35aa4eb0847fa997dd7e3d43 2023-04-19 20:55:19,224 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/b0b7f37b35aa4eb0847fa997dd7e3d43, entries=100, sequenceid=108, filesize=9.3 K 2023-04-19 20:55:19,225 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=138.65 KB/141980 for 5067d98ca5f8d3760c5d0eb2b8bb06dc in 469ms, sequenceid=108, compaction requested=true 2023-04-19 20:55:19,225 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 5067d98ca5f8d3760c5d0eb2b8bb06dc: 2023-04-19 20:55:19,225 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 20:55:19,225 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 20:55:19,225 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 20:55:19,225 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-19 20:55:19,225 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-19 20:55:19,226 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 45303 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-19 20:55:19,226 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 96139 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-19 20:55:19,226 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1912): 5067d98ca5f8d3760c5d0eb2b8bb06dc/B is initiating minor compaction (all files) 2023-04-19 20:55:19,226 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1912): 5067d98ca5f8d3760c5d0eb2b8bb06dc/A is initiating minor compaction (all files) 2023-04-19 20:55:19,226 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5067d98ca5f8d3760c5d0eb2b8bb06dc/B in TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:19,227 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 5067d98ca5f8d3760c5d0eb2b8bb06dc/A in TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:19,227 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/f9c06ee3de7043fb945995fb46b3a40f, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/0de8ee6282ed47b4b49617d785bca139, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/2c76ce6f46344485814067d8447f3a66, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/bb55aa0fba75465fb625cbe11c6eadee] into tmpdir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp, totalSize=44.2 K 2023-04-19 20:55:19,227 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/ad1315a26f8e4150b205bc0be6b2463a, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/c062767b9d634d2fbb9737f52c20e15a, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/c8a8147791b546ac8b0cf5eb79a7cc66, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/f354c78a24c445fd9b41bc8987f232d8] into tmpdir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp, totalSize=93.9 K 2023-04-19 20:55:19,227 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting f9c06ee3de7043fb945995fb46b3a40f, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=49, earliestPutTs=1681937713634 2023-04-19 20:55:19,227 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting ad1315a26f8e4150b205bc0be6b2463a, keycount=150, bloomtype=ROW, size=25.2 K, encoding=NONE, compression=NONE, seqNum=49, earliestPutTs=1681937713634 2023-04-19 20:55:19,227 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 0de8ee6282ed47b4b49617d785bca139, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=85, earliestPutTs=1681937715945 2023-04-19 20:55:19,228 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting c062767b9d634d2fbb9737f52c20e15a, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=85, earliestPutTs=1681937715945 2023-04-19 20:55:19,228 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 2c76ce6f46344485814067d8447f3a66, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=97, earliestPutTs=1681937716576 2023-04-19 20:55:19,228 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting c8a8147791b546ac8b0cf5eb79a7cc66, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=97, earliestPutTs=1681937716576 2023-04-19 20:55:19,228 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting bb55aa0fba75465fb625cbe11c6eadee, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=108, earliestPutTs=1681937718754 2023-04-19 20:55:19,228 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting f354c78a24c445fd9b41bc8987f232d8, keycount=100, bloomtype=ROW, size=18.4 K, encoding=NONE, compression=NONE, seqNum=108, earliestPutTs=1681937718754 2023-04-19 20:55:19,247 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] throttle.PressureAwareThroughputController(145): 5067d98ca5f8d3760c5d0eb2b8bb06dc#B#compaction#318 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 57.69 MB/second 2023-04-19 20:55:19,250 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] throttle.PressureAwareThroughputController(145): 5067d98ca5f8d3760c5d0eb2b8bb06dc#A#compaction#317 average throughput is 3.29 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 57.69 MB/second 2023-04-19 20:55:19,264 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp/B/a807b8737638476cb03aceded8080cc7 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/a807b8737638476cb03aceded8080cc7 2023-04-19 20:55:19,264 DEBUG [Thread-2269] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:19,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:19,267 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:19,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:19,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:19,268 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:19,268 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:19,269 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:19,269 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:19,269 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 5067d98ca5f8d3760c5d0eb2b8bb06dc/B of 5067d98ca5f8d3760c5d0eb2b8bb06dc into a807b8737638476cb03aceded8080cc7(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 20:55:19,269 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5067d98ca5f8d3760c5d0eb2b8bb06dc: 2023-04-19 20:55:19,269 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc., storeName=5067d98ca5f8d3760c5d0eb2b8bb06dc/B, priority=12, startTime=1681937719225; duration=0sec 2023-04-19 20:55:19,269 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 20:55:19,269 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:19,269 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:19,269 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-19 20:55:19,270 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:19,270 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:19,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:19,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:19,271 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:19,271 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:19,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:19,271 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:19,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:19,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:19,271 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:19,272 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:55:19,272 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:19,272 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 45303 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-19 20:55:19,272 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:19,272 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1912): 5067d98ca5f8d3760c5d0eb2b8bb06dc/C is initiating minor compaction (all files) 2023-04-19 20:55:19,272 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:19,272 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:19,272 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5067d98ca5f8d3760c5d0eb2b8bb06dc/C in TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:19,272 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/6d9c33d398a24bbcb7c55c1ee71e29ef, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/977c283471774b328ccb52d7451ffa12, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/adaf97591f734ec69d8eb866392635b5, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/b0b7f37b35aa4eb0847fa997dd7e3d43] into tmpdir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp, totalSize=44.2 K 2023-04-19 20:55:19,272 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:19,272 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:19,272 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 6d9c33d398a24bbcb7c55c1ee71e29ef, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=49, earliestPutTs=1681937713634 2023-04-19 20:55:19,273 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 977c283471774b328ccb52d7451ffa12, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=85, earliestPutTs=1681937715945 2023-04-19 20:55:19,273 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting adaf97591f734ec69d8eb866392635b5, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=97, earliestPutTs=1681937716576 2023-04-19 20:55:19,274 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting b0b7f37b35aa4eb0847fa997dd7e3d43, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=108, earliestPutTs=1681937718754 2023-04-19 20:55:19,276 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:19,276 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. started... 2023-04-19 20:55:19,276 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5067d98ca5f8d3760c5d0eb2b8bb06dc 3/3 column families, dataSize=138.65 KB heapSize=385.77 KB 2023-04-19 20:55:19,276 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5067d98ca5f8d3760c5d0eb2b8bb06dc, store=A 2023-04-19 20:55:19,276 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 20:55:19,276 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5067d98ca5f8d3760c5d0eb2b8bb06dc, store=B 2023-04-19 20:55:19,276 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 20:55:19,277 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5067d98ca5f8d3760c5d0eb2b8bb06dc, store=C 2023-04-19 20:55:19,277 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 20:55:19,289 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] throttle.PressureAwareThroughputController(145): 5067d98ca5f8d3760c5d0eb2b8bb06dc#C#compaction#319 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 57.69 MB/second 2023-04-19 20:55:19,315 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp/C/4be5c3b40ceb4154a4579a3303b110a8 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/4be5c3b40ceb4154a4579a3303b110a8 2023-04-19 20:55:19,319 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 5067d98ca5f8d3760c5d0eb2b8bb06dc/C of 5067d98ca5f8d3760c5d0eb2b8bb06dc into 4be5c3b40ceb4154a4579a3303b110a8(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 20:55:19,319 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5067d98ca5f8d3760c5d0eb2b8bb06dc: 2023-04-19 20:55:19,319 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc., storeName=5067d98ca5f8d3760c5d0eb2b8bb06dc/C, priority=12, startTime=1681937719225; duration=0sec 2023-04-19 20:55:19,319 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 20:55:19,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(9158): Flush requested on 5067d98ca5f8d3760c5d0eb2b8bb06dc 2023-04-19 20:55:19,451 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. as already flushing 2023-04-19 20:55:19,483 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:19,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56610 deadline: 1681937779468, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:19,483 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:19,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56676 deadline: 1681937779468, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:19,494 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:19,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56654 deadline: 1681937779482, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:19,494 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:19,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56670 deadline: 1681937779483, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:19,494 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:19,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56626 deadline: 1681937779483, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:19,587 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:19,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56610 deadline: 1681937779584, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:19,587 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:19,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56676 deadline: 1681937779584, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:19,599 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:19,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56654 deadline: 1681937779595, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:19,599 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:19,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56670 deadline: 1681937779595, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:19,599 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:19,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56626 deadline: 1681937779595, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:19,674 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp/A/79df7df7265d446dbcc5096b9e27a89e as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/79df7df7265d446dbcc5096b9e27a89e 2023-04-19 20:55:19,678 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 5067d98ca5f8d3760c5d0eb2b8bb06dc/A of 5067d98ca5f8d3760c5d0eb2b8bb06dc into 79df7df7265d446dbcc5096b9e27a89e(size=25.4 K), total size for store is 25.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 20:55:19,678 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5067d98ca5f8d3760c5d0eb2b8bb06dc: 2023-04-19 20:55:19,678 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc., storeName=5067d98ca5f8d3760c5d0eb2b8bb06dc/A, priority=12, startTime=1681937719225; duration=0sec 2023-04-19 20:55:19,678 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 20:55:19,719 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304191cbe62fdaba54a37a466b90e837be380 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304191cbe62fdaba54a37a466b90e837be380 2023-04-19 20:55:19,794 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:19,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56676 deadline: 1681937779788, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:19,794 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:19,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56610 deadline: 1681937779788, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:19,807 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:19,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56654 deadline: 1681937779800, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:19,807 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:19,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56670 deadline: 1681937779802, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:19,807 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:19,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56626 deadline: 1681937779802, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:20,103 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:20,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56610 deadline: 1681937780097, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:20,103 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:20,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56676 deadline: 1681937780097, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:20,113 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:20,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56654 deadline: 1681937780108, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:20,113 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:20,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56670 deadline: 1681937780110, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:20,113 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:20,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56626 deadline: 1681937780110, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:20,130 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=144, memsize=49.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp/A/8133d03912154df89d6084ed7259cdcc 2023-04-19 20:55:20,143 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=144 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp/B/81030fc498f446279cd21a78db2b4868 2023-04-19 20:55:20,155 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=144 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp/C/24849ed7896946869e46a41c455c5828 2023-04-19 20:55:20,159 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp/A/8133d03912154df89d6084ed7259cdcc as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/8133d03912154df89d6084ed7259cdcc 2023-04-19 20:55:20,163 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/8133d03912154df89d6084ed7259cdcc, entries=150, sequenceid=144, filesize=25.3 K 2023-04-19 20:55:20,164 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp/B/81030fc498f446279cd21a78db2b4868 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/81030fc498f446279cd21a78db2b4868 2023-04-19 20:55:20,168 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/81030fc498f446279cd21a78db2b4868, entries=150, sequenceid=144, filesize=11.8 K 2023-04-19 20:55:20,169 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp/C/24849ed7896946869e46a41c455c5828 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/24849ed7896946869e46a41c455c5828 2023-04-19 20:55:20,172 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/24849ed7896946869e46a41c455c5828, entries=150, sequenceid=144, filesize=11.8 K 2023-04-19 20:55:20,173 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~138.65 KB/141980, heapSize ~385.72 KB/394976, currentSize=53.67 KB/54960 for 5067d98ca5f8d3760c5d0eb2b8bb06dc in 897ms, sequenceid=144, compaction requested=false 2023-04-19 20:55:20,173 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5067d98ca5f8d3760c5d0eb2b8bb06dc: 2023-04-19 20:55:20,173 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:20,173 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:20,173 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:20,173 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:20,173 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:20,173 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:20,180 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:20,180 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:20,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:20,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:20,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:20,181 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:20,181 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:20,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:20,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:20,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:20,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:20,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:20,182 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:20,182 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@45b0c688[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:20,182 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:20,182 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:20,183 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:20,183 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:20,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:20,184 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:20,184 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:20,184 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:20,184 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:20,184 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:20,185 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:20,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:20,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:20,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:20,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:20,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:20,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:20,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:20,186 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:20,186 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:20,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:20,186 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:20,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:20,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:20,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:20,187 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:20,187 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:20,188 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:20,188 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:20,188 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:20,188 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:20,189 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:20,189 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:20,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:20,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:20,189 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:20,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:20,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:20,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:20,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:20,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:20,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:20,190 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:20,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:20,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:20,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:20,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:20,193 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:20,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:20,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:20,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:20,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:20,208 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:20,208 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:55:20,208 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:20,208 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:20,209 DEBUG [Thread-2269] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:20,209 DEBUG [Thread-2269] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:20,211 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:20,211 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:20,211 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:20,211 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:20,211 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:20,211 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:20,211 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:20,211 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:20,211 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:20,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:20,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:20,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:20,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:20,309 DEBUG [Thread-2269] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:20,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:20,311 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:20,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:20,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:20,313 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:20,313 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:20,313 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:20,313 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:20,313 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:20,313 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:20,314 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:20,314 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:20,315 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:20,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:20,315 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:20,315 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:20,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:20,315 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:20,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:20,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:20,316 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:20,316 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:55:20,316 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:20,316 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:20,316 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:20,316 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:20,316 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:20,316 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:20,316 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:20,316 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. started... 2023-04-19 20:55:20,316 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5067d98ca5f8d3760c5d0eb2b8bb06dc 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-19 20:55:20,317 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5067d98ca5f8d3760c5d0eb2b8bb06dc, store=A 2023-04-19 20:55:20,317 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:55:20,317 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5067d98ca5f8d3760c5d0eb2b8bb06dc, store=B 2023-04-19 20:55:20,317 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:55:20,317 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5067d98ca5f8d3760c5d0eb2b8bb06dc, store=C 2023-04-19 20:55:20,317 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:55:20,332 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230419e8b4ed14982541618d6cbe67c38810c0 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419e8b4ed14982541618d6cbe67c38810c0 2023-04-19 20:55:20,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(9158): Flush requested on 5067d98ca5f8d3760c5d0eb2b8bb06dc 2023-04-19 20:55:20,606 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. as already flushing 2023-04-19 20:55:20,645 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:20,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56626 deadline: 1681937780635, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:20,645 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:20,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56610 deadline: 1681937780636, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:20,646 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681937720628, max timestamp=1681937720644Num uniques 859; 2023-04-19 20:55:20,646 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-19 20:55:20,646 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681937720628, max timestamp=1681937720644Num uniques 859; 2023-04-19 20:55:20,646 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-19 20:55:20,647 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681937720628, max timestamp=1681937720644Num uniques 859; 2023-04-19 20:55:20,647 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-19 20:55:20,684 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:20,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56654 deadline: 1681937780675, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:20,685 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:20,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56670 deadline: 1681937780682, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:20,689 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:20,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56676 deadline: 1681937780684, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:20,737 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=157, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp/A/98ee35ec3c42466faf36f0d83c221698 2023-04-19 20:55:20,750 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=157 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp/B/71243cd9934b49af886a82cb5d787d7b 2023-04-19 20:55:20,753 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:20,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56626 deadline: 1681937780746, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:20,754 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:20,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56610 deadline: 1681937780746, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:20,791 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:20,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56654 deadline: 1681937780785, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:20,791 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:20,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56670 deadline: 1681937780786, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:20,791 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:20,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56676 deadline: 1681937780790, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:20,960 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:20,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56626 deadline: 1681937780955, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:20,961 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:20,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56610 deadline: 1681937780956, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:20,998 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:20,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56670 deadline: 1681937780992, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:20,998 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:20,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56654 deadline: 1681937780992, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:20,998 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:20,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56676 deadline: 1681937780994, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:21,174 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=157 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp/C/88399863f67946f4b492e08c77381c49 2023-04-19 20:55:21,179 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp/A/98ee35ec3c42466faf36f0d83c221698 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/98ee35ec3c42466faf36f0d83c221698 2023-04-19 20:55:21,183 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/98ee35ec3c42466faf36f0d83c221698, entries=150, sequenceid=157, filesize=25.3 K 2023-04-19 20:55:21,185 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp/B/71243cd9934b49af886a82cb5d787d7b as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/71243cd9934b49af886a82cb5d787d7b 2023-04-19 20:55:21,189 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/71243cd9934b49af886a82cb5d787d7b, entries=150, sequenceid=157, filesize=11.8 K 2023-04-19 20:55:21,190 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp/C/88399863f67946f4b492e08c77381c49 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/88399863f67946f4b492e08c77381c49 2023-04-19 20:55:21,195 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/88399863f67946f4b492e08c77381c49, entries=150, sequenceid=157, filesize=11.8 K 2023-04-19 20:55:21,195 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 5067d98ca5f8d3760c5d0eb2b8bb06dc in 879ms, sequenceid=157, compaction requested=true 2023-04-19 20:55:21,196 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5067d98ca5f8d3760c5d0eb2b8bb06dc: 2023-04-19 20:55:21,196 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:21,196 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:21,196 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:21,196 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:21,196 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:21,196 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:21,198 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:21,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:21,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:21,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:21,198 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:21,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:21,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:21,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:21,199 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:21,199 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:21,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:21,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:21,200 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:21,200 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3c4e5b59[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:21,200 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:21,201 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:21,202 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:21,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:21,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:21,202 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:21,202 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:21,202 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:21,202 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:21,202 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:21,205 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:21,205 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:21,205 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:21,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:21,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:21,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:21,205 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:21,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:21,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:21,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:21,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:21,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:21,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:21,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:21,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:21,207 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:21,207 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:21,207 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:21,207 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:21,207 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:21,207 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:21,209 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:21,209 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:21,209 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:21,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:21,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:21,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:21,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:21,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:21,210 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:21,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:21,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:21,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:21,210 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:21,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:21,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:21,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:21,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:21,213 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:21,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:21,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:21,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:21,215 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:21,215 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:21,215 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:21,215 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:21,215 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:21,215 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:55:21,215 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:21,215 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:21,215 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:21,215 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:21,215 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:21,215 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:21,215 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:21,215 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:21,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:21,216 DEBUG [Thread-2269] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:21,216 DEBUG [Thread-2269] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:21,216 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:21,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:21,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(9158): Flush requested on 5067d98ca5f8d3760c5d0eb2b8bb06dc 2023-04-19 20:55:21,268 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 5067d98ca5f8d3760c5d0eb2b8bb06dc 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-04-19 20:55:21,268 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5067d98ca5f8d3760c5d0eb2b8bb06dc, store=A 2023-04-19 20:55:21,268 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 20:55:21,268 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5067d98ca5f8d3760c5d0eb2b8bb06dc, store=B 2023-04-19 20:55:21,269 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 20:55:21,269 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5067d98ca5f8d3760c5d0eb2b8bb06dc, store=C 2023-04-19 20:55:21,269 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 20:55:21,293 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041914b132754ce14bffac33c7e4d6f46007 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041914b132754ce14bffac33c7e4d6f46007 2023-04-19 20:55:21,305 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:21,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56626 deadline: 1681937781298, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:21,309 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:21,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56654 deadline: 1681937781300, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:21,309 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:21,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56676 deadline: 1681937781300, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:21,310 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:21,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56670 deadline: 1681937781301, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:21,310 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:21,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56610 deadline: 1681937781305, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:21,316 DEBUG [Thread-2269] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:21,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:21,318 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:21,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:21,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:21,319 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:21,319 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:21,319 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:21,319 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:21,320 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:21,320 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:21,321 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:21,321 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:21,321 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:21,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:21,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:21,324 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:21,324 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:21,324 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:21,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:21,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:21,325 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:21,325 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:55:21,325 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:21,325 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:21,325 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:21,325 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:21,331 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:21,331 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:21,331 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:21,331 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. started... 2023-04-19 20:55:21,331 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. as already flushing 2023-04-19 20:55:21,331 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:21,331 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:21,331 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:21,331 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:21,331 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:21,331 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:21,334 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:21,334 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:21,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:21,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:21,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:21,334 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:21,334 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:21,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:21,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:21,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:21,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:21,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:21,336 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:21,336 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@43768155[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:21,336 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:21,336 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:21,337 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:21,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:21,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:21,338 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:21,338 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:21,338 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:21,338 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:21,338 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:21,339 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:21,339 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:21,339 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:21,339 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:21,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:21,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:21,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:21,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:21,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:21,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:21,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:21,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:21,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:21,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:21,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:21,341 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:21,341 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:21,342 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:21,342 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:21,342 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:21,342 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:21,343 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:21,343 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:21,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:21,343 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:21,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:21,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:21,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:21,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:21,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:21,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:21,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:21,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:21,344 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:21,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:21,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:21,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:21,345 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:21,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:21,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:21,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:21,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:21,349 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:21,349 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:21,349 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:21,349 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:55:21,349 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:21,349 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:21,349 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:21,349 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:21,349 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:21,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:21,349 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:21,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:21,349 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:21,349 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:21,349 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:21,349 DEBUG [Thread-2269] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:21,349 DEBUG [Thread-2269] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:21,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:21,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:21,409 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:21,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56626 deadline: 1681937781406, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:21,419 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:21,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56610 deadline: 1681937781411, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:21,450 DEBUG [Thread-2269] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:21,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:21,452 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:21,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:21,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:21,456 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:21,456 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:21,456 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:21,456 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:21,456 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:21,456 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:21,458 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:21,458 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:21,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:21,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:21,458 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:21,458 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:21,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:21,459 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:21,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:21,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:21,459 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:21,459 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:55:21,459 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:21,459 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:21,459 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:21,459 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:21,460 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:21,460 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:21,460 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:21,460 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. started... 2023-04-19 20:55:21,460 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. as already flushing 2023-04-19 20:55:21,460 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:21,460 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:21,460 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:21,460 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:21,460 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:21,460 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:21,461 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:21,461 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:21,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:21,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:21,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:21,462 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:21,462 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:21,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:21,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:21,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:21,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:21,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:21,463 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:21,463 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@157586b[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:21,463 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:21,463 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:21,464 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:21,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:21,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:21,465 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:21,465 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:21,465 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:21,465 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:21,465 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:21,467 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:21,467 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:21,467 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:21,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:21,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:21,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:21,467 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:21,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:21,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:21,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:21,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:21,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:21,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:21,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:21,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:21,470 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:21,470 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:21,470 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:21,470 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:21,470 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:21,470 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:21,472 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:21,472 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:21,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:21,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:21,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:21,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:21,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:21,472 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:21,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:21,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:21,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:21,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:21,473 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:21,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:21,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:21,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:21,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:21,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:21,476 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:21,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:21,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:21,481 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:21,481 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:21,481 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:21,481 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:21,481 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:21,481 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:21,481 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:21,481 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:21,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:21,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:21,481 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:21,481 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:21,481 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:21,481 DEBUG [Thread-2269] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:21,481 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:21,482 DEBUG [Thread-2269] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:21,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:21,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:21,582 DEBUG [Thread-2269] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:21,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:21,584 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:21,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:21,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:21,585 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:21,585 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:21,585 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:21,585 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:21,586 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:21,586 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:21,587 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:21,587 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:21,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:21,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:21,588 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:21,588 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:21,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:21,588 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:21,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:21,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:21,588 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:21,590 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:55:21,590 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:21,590 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:21,590 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:21,590 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:21,590 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:21,590 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:21,590 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:21,590 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. started... 2023-04-19 20:55:21,591 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. as already flushing 2023-04-19 20:55:21,591 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:21,591 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:21,591 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:21,591 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:21,591 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:21,591 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:21,592 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:21,592 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:21,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:21,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:21,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:21,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:21,593 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:21,593 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:21,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:21,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:21,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:21,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:21,594 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:21,594 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@42904a4f[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:21,594 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:21,594 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:21,595 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:21,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:21,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:21,595 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:21,595 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:21,595 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:21,595 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:21,595 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:21,597 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:21,597 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:21,597 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:21,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:21,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:21,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:21,597 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:21,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:21,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:21,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:21,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:21,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:21,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:21,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:21,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:21,599 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:21,599 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:21,599 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:21,599 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:21,599 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:21,599 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:21,601 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:21,601 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:21,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:21,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:21,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:21,601 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:21,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:21,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:21,602 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:21,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:21,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:21,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:21,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:21,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:21,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:21,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:21,603 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:21,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:21,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:21,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:21,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:21,605 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:21,605 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:21,605 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:21,605 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:21,605 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:21,605 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:21,605 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:21,605 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:21,605 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:21,605 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:21,605 DEBUG [Thread-2269] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:21,605 DEBUG [Thread-2269] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:21,606 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:21,606 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:21,606 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:21,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:21,606 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:55:21,606 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:21,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:21,617 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:21,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56626 deadline: 1681937781611, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:21,630 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:21,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56610 deadline: 1681937781621, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:21,703 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=193, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp/A/a373fc3f0b68496894e5b98c7b87c4a2 2023-04-19 20:55:21,705 DEBUG [Thread-2269] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:21,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:21,708 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:21,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:21,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:21,713 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:21,713 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:21,713 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:21,713 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:21,714 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:21,714 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:21,715 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:21,715 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:21,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:21,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:21,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:21,715 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:21,715 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:21,716 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:21,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:21,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:21,716 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:21,720 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:55:21,720 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:21,720 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:21,720 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:21,720 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:21,720 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:21,720 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:21,720 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:21,720 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. started... 2023-04-19 20:55:21,720 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. as already flushing 2023-04-19 20:55:21,721 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:21,721 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:21,721 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:21,721 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:21,721 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:21,721 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:21,722 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:21,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:21,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:21,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:21,722 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:21,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:21,722 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:21,722 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:21,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:21,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:21,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:21,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:21,723 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:21,724 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@cca55d7[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:21,724 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:21,724 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:21,727 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:21,727 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:21,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:21,727 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:21,727 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:21,728 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:21,730 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:21,730 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:21,730 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:21,730 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:21,730 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:21,730 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:21,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:21,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:21,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:21,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:21,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:21,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:21,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:21,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:21,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:21,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:21,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:21,732 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:21,732 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:21,732 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:21,732 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:21,732 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:21,732 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:21,733 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:21,733 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:21,733 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:21,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:21,733 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:21,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:21,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:21,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:21,734 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:21,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:21,734 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:21,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:21,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:21,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:21,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:21,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:21,735 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:21,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:21,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:21,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:21,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:21,737 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:21,737 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:21,737 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:21,737 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:21,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:21,737 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:21,737 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:21,737 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:21,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:21,738 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:21,738 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:21,738 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:21,738 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:21,738 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:21,738 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:21,738 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:21,738 DEBUG [Thread-2269] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:21,738 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:55:21,738 DEBUG [Thread-2269] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:21,815 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:21,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56654 deadline: 1681937781811, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:21,818 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:21,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56676 deadline: 1681937781814, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:21,818 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:21,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56670 deadline: 1681937781814, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:21,838 DEBUG [Thread-2269] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:21,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:21,841 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:21,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:21,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:21,842 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:21,842 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:21,842 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:21,842 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:21,842 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:21,842 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:21,844 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:21,844 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:21,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:21,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:21,845 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:21,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:21,845 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:21,845 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:21,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:21,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:21,845 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:21,846 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:55:21,846 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:21,846 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:21,846 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:21,846 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:21,846 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:21,846 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:21,846 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:21,846 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. started... 2023-04-19 20:55:21,846 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. as already flushing 2023-04-19 20:55:21,846 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:21,846 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:21,846 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:21,846 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:21,846 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:21,846 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:21,848 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:21,848 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:21,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:21,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:21,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:21,848 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:21,848 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:21,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:21,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:21,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:21,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:21,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:21,850 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:21,850 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7b410132[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:21,850 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:21,850 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:21,851 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:21,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:21,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:21,851 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:21,851 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:21,851 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:21,851 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:21,851 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:21,853 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:21,853 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:21,853 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:21,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:21,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:21,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:21,853 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:21,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:21,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:21,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:21,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:21,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:21,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:21,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:21,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:21,855 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:21,855 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:21,855 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:21,855 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:21,855 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:21,855 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:21,859 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:21,859 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:21,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:21,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:21,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:21,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:21,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:21,859 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:21,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:21,859 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:21,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:21,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:21,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:21,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:21,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:21,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:21,860 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:21,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:21,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:21,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:21,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:21,863 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:21,863 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:21,863 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:21,863 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:21,863 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:55:21,863 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:21,863 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:21,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:21,863 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:21,863 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:21,863 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:21,863 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:21,863 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:21,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:21,863 DEBUG [Thread-2269] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:21,863 DEBUG [Thread-2269] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:21,863 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:21,863 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:21,864 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:21,925 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:21,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56626 deadline: 1681937781920, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:21,939 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:21,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56610 deadline: 1681937781933, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:21,963 DEBUG [Thread-2269] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:21,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:21,965 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:21,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:21,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:21,967 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:21,967 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:21,967 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:21,967 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:21,968 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:21,968 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:21,972 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:21,972 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:21,972 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:21,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:21,972 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:21,972 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:21,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:21,973 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:21,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:21,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:21,973 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:21,973 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:55:21,973 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:21,974 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:21,974 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:21,974 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:21,974 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:21,974 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:21,974 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:21,974 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. started... 2023-04-19 20:55:21,974 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. as already flushing 2023-04-19 20:55:21,974 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:21,974 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:21,974 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:21,974 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:21,974 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:21,974 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:21,979 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:21,979 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:21,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:21,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:21,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:21,979 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:21,979 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:21,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:21,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:21,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:21,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:21,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:21,981 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:21,981 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@40ee8a31[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:21,981 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:21,981 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:21,982 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:21,982 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:21,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:21,982 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:21,982 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:21,982 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:21,982 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:21,982 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:21,984 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:21,984 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:21,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:21,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:21,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:21,984 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:21,984 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:21,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:21,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:21,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:21,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:21,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:21,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:21,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:21,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:21,986 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:21,986 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:21,986 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:21,986 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:21,986 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:21,986 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:21,987 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:21,988 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:21,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:21,988 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:21,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:21,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:21,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:21,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:21,988 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:21,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:21,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:21,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:21,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:21,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:21,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:21,990 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:21,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:21,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:21,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:21,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:21,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:21,993 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:21,993 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:21,993 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:21,993 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:21,993 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:21,993 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:21,993 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:21,994 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:21,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:21,993 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:21,994 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:21,994 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:21,994 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:21,994 DEBUG [Thread-2269] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:21,994 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:55:21,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:21,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:21,994 DEBUG [Thread-2269] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:21,994 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:22,094 DEBUG [Thread-2269] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:22,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:22,097 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:22,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:22,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:22,098 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:22,098 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:22,098 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:22,098 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:22,099 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:22,099 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:22,100 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:22,100 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:22,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:22,100 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:22,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:22,100 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:22,101 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:22,101 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:22,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:22,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:22,101 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:22,101 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:55:22,102 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:22,102 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:22,102 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:22,102 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:22,102 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:22,102 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:22,102 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:22,102 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. started... 2023-04-19 20:55:22,102 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. as already flushing 2023-04-19 20:55:22,102 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:22,102 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:22,102 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:22,102 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:22,102 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:22,102 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:22,104 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:22,104 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:22,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:22,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:22,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:22,104 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:22,104 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:22,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:22,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:22,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:22,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:22,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:22,105 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:22,105 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6c6251[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:22,105 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:22,105 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:22,107 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:22,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:22,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:22,107 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:22,107 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:22,107 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:22,107 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:22,107 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:22,109 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:22,109 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:22,109 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:22,109 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:22,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:22,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:22,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:22,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:22,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:22,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:22,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:22,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:22,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:22,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:22,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:22,112 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:22,112 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:22,112 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:22,112 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:22,112 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:22,112 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:22,114 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:22,114 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:22,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:22,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:22,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:22,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:22,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:22,114 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:22,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:22,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:22,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:22,115 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:22,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:22,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:22,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:22,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:22,116 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:22,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:22,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:22,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:22,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:22,117 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:22,118 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:22,118 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:22,118 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:22,118 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:22,118 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:22,118 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:22,118 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:22,118 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:22,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:22,118 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:22,118 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:22,118 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=193 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp/B/842f3fe790614916a9dfc6c04c3007ad 2023-04-19 20:55:22,118 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:22,118 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:22,118 DEBUG [Thread-2269] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:22,118 DEBUG [Thread-2269] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:22,119 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:55:22,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:22,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:22,219 DEBUG [Thread-2269] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:22,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:22,221 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:22,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:22,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:22,222 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:22,222 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:22,222 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:22,222 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:22,223 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:22,223 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:22,224 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:22,224 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:22,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:22,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:22,224 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:22,224 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:22,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:22,225 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:22,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:22,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:22,225 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:22,225 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:22,225 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:22,225 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:22,225 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:22,235 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:22,235 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:22,235 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:22,235 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. started... 2023-04-19 20:55:22,235 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. as already flushing 2023-04-19 20:55:22,235 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:22,235 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:22,235 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:22,235 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:22,235 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:22,236 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:22,238 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:22,238 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:22,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:22,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:22,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:22,238 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:22,238 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:22,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:22,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:22,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:22,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:22,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:22,240 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:22,240 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5e56fb66[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:22,240 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:22,240 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:22,241 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:22,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:22,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:22,241 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:22,241 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:22,241 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:22,242 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:22,242 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:22,243 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:22,243 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:22,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:22,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:22,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:22,243 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:22,243 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:22,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:22,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:22,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:22,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:22,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:22,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:22,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:22,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:22,245 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:22,245 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:22,245 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:22,245 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:22,245 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:22,245 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:22,247 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:22,247 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:22,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:22,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:22,247 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:22,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:22,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:22,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:22,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:22,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:22,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:22,248 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:22,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:22,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:22,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:22,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:22,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:22,249 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:22,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:22,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:22,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:22,251 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:22,251 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:22,251 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:22,251 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:22,251 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:22,251 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:22,251 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:22,251 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:22,252 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:22,251 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:22,251 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:22,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:22,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:22,252 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:22,252 DEBUG [Thread-2269] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:22,252 DEBUG [Thread-2269] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:22,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:22,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:22,352 DEBUG [Thread-2269] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:22,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:22,354 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:22,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:22,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:22,356 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:22,356 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:22,356 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:22,356 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:22,358 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:22,358 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:22,359 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:22,359 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:22,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:22,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:22,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:22,359 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:22,359 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:22,360 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:22,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:22,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:22,360 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:22,361 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:55:22,361 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:22,361 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:22,361 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:22,361 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:22,368 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:22,368 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:22,368 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:22,368 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. started... 2023-04-19 20:55:22,368 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. as already flushing 2023-04-19 20:55:22,368 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:22,368 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:22,368 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:22,368 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:22,368 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:22,368 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:22,374 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:22,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:22,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:22,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:22,374 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:22,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:22,374 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:22,374 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:22,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:22,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:22,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:22,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:22,375 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:22,375 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2ded390f[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:22,375 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:22,375 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:22,377 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:22,377 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:22,377 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:22,377 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:22,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:22,377 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:22,377 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:22,377 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:22,378 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:22,379 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:22,378 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:22,379 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:22,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:22,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:22,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:22,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:22,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:22,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:22,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:22,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:22,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:22,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:22,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:22,383 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:22,383 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:22,383 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:22,383 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:22,383 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:22,383 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:22,385 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:22,385 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:22,385 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:22,385 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:22,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:22,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:22,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:22,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:22,385 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:22,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:22,385 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:22,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:22,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:22,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:22,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:22,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:22,386 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:22,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:22,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:22,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:22,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:22,390 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:22,390 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:22,390 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:22,390 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:22,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:22,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:22,390 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:22,390 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:22,390 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:22,390 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:55:22,390 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:22,390 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:22,390 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:22,390 DEBUG [Thread-2269] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:22,390 DEBUG [Thread-2269] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:22,390 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:22,390 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:22,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:22,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:22,433 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:22,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56626 deadline: 1681937782427, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:22,450 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:22,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56610 deadline: 1681937782443, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:22,490 DEBUG [Thread-2269] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:22,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:22,493 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:22,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:22,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:22,494 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:22,494 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:22,494 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:22,494 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:22,494 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:22,494 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:22,495 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:22,495 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:22,496 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:22,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:22,496 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:22,496 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:22,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:22,496 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:22,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:22,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:22,497 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:22,497 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:55:22,497 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:22,498 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:22,498 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:22,498 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:22,498 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:22,498 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:22,498 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:22,498 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. started... 2023-04-19 20:55:22,498 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. as already flushing 2023-04-19 20:55:22,498 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:22,498 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:22,498 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:22,498 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:22,498 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:22,498 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:22,504 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:22,504 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:22,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:22,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:22,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:22,504 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:22,504 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:22,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:22,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:22,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:22,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:22,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:22,506 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:22,506 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@19e68181[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:22,506 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:22,506 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:22,507 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:22,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:22,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:22,507 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:22,507 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:22,507 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:22,507 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:22,507 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:22,509 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:22,509 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:22,509 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:22,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:22,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:22,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:22,509 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:22,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:22,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:22,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:22,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:22,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:22,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:22,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:22,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:22,511 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:22,511 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:22,511 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:22,511 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:22,511 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:22,511 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:22,513 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:22,513 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:22,513 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:22,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:22,513 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:22,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:22,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:22,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:22,513 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:22,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:22,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:22,513 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:22,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:22,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:22,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:22,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:22,515 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:22,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:22,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:22,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:22,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:22,518 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:22,518 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:22,518 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:22,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:22,518 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:22,518 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:22,518 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:22,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:22,518 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:22,518 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:55:22,518 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:22,518 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:22,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:22,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:22,518 DEBUG [Thread-2269] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:22,518 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:22,518 DEBUG [Thread-2269] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:22,518 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:22,519 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:22,529 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=193 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp/C/7ffd3c4c8f9a4b26b30d9ff4b109fcca 2023-04-19 20:55:22,533 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp/A/a373fc3f0b68496894e5b98c7b87c4a2 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/a373fc3f0b68496894e5b98c7b87c4a2 2023-04-19 20:55:22,536 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/a373fc3f0b68496894e5b98c7b87c4a2, entries=200, sequenceid=193, filesize=32.1 K 2023-04-19 20:55:22,537 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp/B/842f3fe790614916a9dfc6c04c3007ad as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/842f3fe790614916a9dfc6c04c3007ad 2023-04-19 20:55:22,541 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/842f3fe790614916a9dfc6c04c3007ad, entries=150, sequenceid=193, filesize=11.8 K 2023-04-19 20:55:22,541 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp/C/7ffd3c4c8f9a4b26b30d9ff4b109fcca as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/7ffd3c4c8f9a4b26b30d9ff4b109fcca 2023-04-19 20:55:22,545 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/7ffd3c4c8f9a4b26b30d9ff4b109fcca, entries=150, sequenceid=193, filesize=11.8 K 2023-04-19 20:55:22,546 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=46.96 KB/48090 for 5067d98ca5f8d3760c5d0eb2b8bb06dc in 1278ms, sequenceid=193, compaction requested=true 2023-04-19 20:55:22,546 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 5067d98ca5f8d3760c5d0eb2b8bb06dc: 2023-04-19 20:55:22,546 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 20:55:22,546 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-19 20:55:22,546 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-19 20:55:22,546 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 20:55:22,546 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 20:55:22,547 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 110664 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-19 20:55:22,547 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1912): 5067d98ca5f8d3760c5d0eb2b8bb06dc/A is initiating minor compaction (all files) 2023-04-19 20:55:22,547 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48234 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-19 20:55:22,548 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 5067d98ca5f8d3760c5d0eb2b8bb06dc/A in TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:22,548 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1912): 5067d98ca5f8d3760c5d0eb2b8bb06dc/B is initiating minor compaction (all files) 2023-04-19 20:55:22,548 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/79df7df7265d446dbcc5096b9e27a89e, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/8133d03912154df89d6084ed7259cdcc, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/98ee35ec3c42466faf36f0d83c221698, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/a373fc3f0b68496894e5b98c7b87c4a2] into tmpdir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp, totalSize=108.1 K 2023-04-19 20:55:22,548 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5067d98ca5f8d3760c5d0eb2b8bb06dc/B in TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:22,548 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/a807b8737638476cb03aceded8080cc7, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/81030fc498f446279cd21a78db2b4868, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/71243cd9934b49af886a82cb5d787d7b, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/842f3fe790614916a9dfc6c04c3007ad] into tmpdir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp, totalSize=47.1 K 2023-04-19 20:55:22,548 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 79df7df7265d446dbcc5096b9e27a89e, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=108, earliestPutTs=1681937716578 2023-04-19 20:55:22,549 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting a807b8737638476cb03aceded8080cc7, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=108, earliestPutTs=1681937716578 2023-04-19 20:55:22,549 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 8133d03912154df89d6084ed7259cdcc, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=144, earliestPutTs=1681937718819 2023-04-19 20:55:22,549 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 81030fc498f446279cd21a78db2b4868, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=144, earliestPutTs=1681937718819 2023-04-19 20:55:22,549 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 98ee35ec3c42466faf36f0d83c221698, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=157, earliestPutTs=1681937719467 2023-04-19 20:55:22,549 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 71243cd9934b49af886a82cb5d787d7b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=157, earliestPutTs=1681937719467 2023-04-19 20:55:22,549 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting a373fc3f0b68496894e5b98c7b87c4a2, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=193, earliestPutTs=1681937720674 2023-04-19 20:55:22,550 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 842f3fe790614916a9dfc6c04c3007ad, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=193, earliestPutTs=1681937720674 2023-04-19 20:55:22,578 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] throttle.PressureAwareThroughputController(145): 5067d98ca5f8d3760c5d0eb2b8bb06dc#B#compaction#329 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 57.69 MB/second 2023-04-19 20:55:22,598 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] throttle.PressureAwareThroughputController(145): 5067d98ca5f8d3760c5d0eb2b8bb06dc#A#compaction#330 average throughput is 0.99 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 57.69 MB/second 2023-04-19 20:55:22,619 DEBUG [Thread-2269] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:22,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:22,624 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:22,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:22,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:22,628 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:22,628 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:22,628 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:22,628 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:22,629 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:22,629 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:22,630 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:22,630 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:22,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:22,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:22,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:22,631 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:22,631 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:22,631 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:22,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:22,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:22,631 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:22,632 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:55:22,632 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:22,632 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:22,632 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:22,632 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:22,632 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:22,632 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:22,632 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:22,632 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. started... 2023-04-19 20:55:22,632 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5067d98ca5f8d3760c5d0eb2b8bb06dc 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-19 20:55:22,632 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5067d98ca5f8d3760c5d0eb2b8bb06dc, store=A 2023-04-19 20:55:22,632 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:55:22,632 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5067d98ca5f8d3760c5d0eb2b8bb06dc, store=B 2023-04-19 20:55:22,632 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:55:22,633 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5067d98ca5f8d3760c5d0eb2b8bb06dc, store=C 2023-04-19 20:55:22,633 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:55:22,649 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230419edf0c10320be40c3a4f09a94b81e613f to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419edf0c10320be40c3a4f09a94b81e613f 2023-04-19 20:55:22,655 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=203, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp/A/ce774494e7894fbab682540dbcbb5957 2023-04-19 20:55:22,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(9158): Flush requested on 5067d98ca5f8d3760c5d0eb2b8bb06dc 2023-04-19 20:55:22,822 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. as already flushing 2023-04-19 20:55:22,867 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681937722850, max timestamp=1681937722865Num uniques 859; 2023-04-19 20:55:22,867 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681937722850, max timestamp=1681937722865Num uniques 859; 2023-04-19 20:55:22,867 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681937722850, max timestamp=1681937722865Num uniques 859; 2023-04-19 20:55:22,868 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 20:55:22,867 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 20:55:22,868 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 20:55:22,912 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:22,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56676 deadline: 1681937782900, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:22,917 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:22,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56654 deadline: 1681937782908, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:22,917 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:22,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56670 deadline: 1681937782912, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:23,006 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp/B/1fabe96869f7467aa6c4c29409fb6345 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/1fabe96869f7467aa6c4c29409fb6345 2023-04-19 20:55:23,016 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 5067d98ca5f8d3760c5d0eb2b8bb06dc/B of 5067d98ca5f8d3760c5d0eb2b8bb06dc into 1fabe96869f7467aa6c4c29409fb6345(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 20:55:23,017 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5067d98ca5f8d3760c5d0eb2b8bb06dc: 2023-04-19 20:55:23,017 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc., storeName=5067d98ca5f8d3760c5d0eb2b8bb06dc/B, priority=12, startTime=1681937722546; duration=0sec 2023-04-19 20:55:23,017 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 20:55:23,017 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-19 20:55:23,018 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48234 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-19 20:55:23,018 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:23,018 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1912): 5067d98ca5f8d3760c5d0eb2b8bb06dc/C is initiating minor compaction (all files) 2023-04-19 20:55:23,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56676 deadline: 1681937783014, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:23,018 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5067d98ca5f8d3760c5d0eb2b8bb06dc/C in TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:23,018 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/4be5c3b40ceb4154a4579a3303b110a8, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/24849ed7896946869e46a41c455c5828, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/88399863f67946f4b492e08c77381c49, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/7ffd3c4c8f9a4b26b30d9ff4b109fcca] into tmpdir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp, totalSize=47.1 K 2023-04-19 20:55:23,019 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 4be5c3b40ceb4154a4579a3303b110a8, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=108, earliestPutTs=1681937716578 2023-04-19 20:55:23,019 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 24849ed7896946869e46a41c455c5828, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=144, earliestPutTs=1681937718819 2023-04-19 20:55:23,020 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 88399863f67946f4b492e08c77381c49, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=157, earliestPutTs=1681937719467 2023-04-19 20:55:23,020 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 7ffd3c4c8f9a4b26b30d9ff4b109fcca, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=193, earliestPutTs=1681937720674 2023-04-19 20:55:23,030 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] throttle.PressureAwareThroughputController(145): 5067d98ca5f8d3760c5d0eb2b8bb06dc#C#compaction#333 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 57.69 MB/second 2023-04-19 20:55:23,032 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:23,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56670 deadline: 1681937783025, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:23,033 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:23,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56654 deadline: 1681937783025, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:23,037 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp/A/3e5713ebc792425caf9af7d5349a2857 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/3e5713ebc792425caf9af7d5349a2857 2023-04-19 20:55:23,047 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 5067d98ca5f8d3760c5d0eb2b8bb06dc/A of 5067d98ca5f8d3760c5d0eb2b8bb06dc into 3e5713ebc792425caf9af7d5349a2857(size=25.7 K), total size for store is 25.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 20:55:23,047 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5067d98ca5f8d3760c5d0eb2b8bb06dc: 2023-04-19 20:55:23,047 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc., storeName=5067d98ca5f8d3760c5d0eb2b8bb06dc/A, priority=12, startTime=1681937722546; duration=0sec 2023-04-19 20:55:23,047 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 20:55:23,047 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp/C/8ec186b6992642ffaafc818130f98183 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/8ec186b6992642ffaafc818130f98183 2023-04-19 20:55:23,052 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 5067d98ca5f8d3760c5d0eb2b8bb06dc/C of 5067d98ca5f8d3760c5d0eb2b8bb06dc into 8ec186b6992642ffaafc818130f98183(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 20:55:23,052 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5067d98ca5f8d3760c5d0eb2b8bb06dc: 2023-04-19 20:55:23,052 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc., storeName=5067d98ca5f8d3760c5d0eb2b8bb06dc/C, priority=12, startTime=1681937722546; duration=0sec 2023-04-19 20:55:23,052 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 20:55:23,067 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=203 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp/B/af1bd3ce6e9345748004cbb754a4efe6 2023-04-19 20:55:23,226 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:23,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56676 deadline: 1681937783221, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:23,236 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:23,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56654 deadline: 1681937783234, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:23,237 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:23,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56670 deadline: 1681937783234, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:23,446 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:23,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56626 deadline: 1681937783442, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:23,458 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:23,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56610 deadline: 1681937783454, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:23,476 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=203 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp/C/68037ac8c97b40a4be643b017208beeb 2023-04-19 20:55:23,480 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp/A/ce774494e7894fbab682540dbcbb5957 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/ce774494e7894fbab682540dbcbb5957 2023-04-19 20:55:23,483 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/ce774494e7894fbab682540dbcbb5957, entries=150, sequenceid=203, filesize=25.3 K 2023-04-19 20:55:23,484 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp/B/af1bd3ce6e9345748004cbb754a4efe6 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/af1bd3ce6e9345748004cbb754a4efe6 2023-04-19 20:55:23,487 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/af1bd3ce6e9345748004cbb754a4efe6, entries=150, sequenceid=203, filesize=11.8 K 2023-04-19 20:55:23,487 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp/C/68037ac8c97b40a4be643b017208beeb as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/68037ac8c97b40a4be643b017208beeb 2023-04-19 20:55:23,490 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/68037ac8c97b40a4be643b017208beeb, entries=150, sequenceid=203, filesize=11.8 K 2023-04-19 20:55:23,491 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 5067d98ca5f8d3760c5d0eb2b8bb06dc in 859ms, sequenceid=203, compaction requested=false 2023-04-19 20:55:23,492 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5067d98ca5f8d3760c5d0eb2b8bb06dc: 2023-04-19 20:55:23,492 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:23,492 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:23,492 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:23,492 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:23,492 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:23,492 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:23,494 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:23,494 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:23,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:23,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:23,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:23,494 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:23,494 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:23,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:23,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:23,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:23,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:23,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:23,495 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:23,495 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7a7b0606[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:23,495 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:23,495 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:23,496 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:23,496 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:23,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:23,497 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:23,497 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:23,497 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:23,497 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:23,497 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:23,499 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:23,499 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:23,499 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:23,499 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:23,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:23,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:23,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:23,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:23,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:23,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:23,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:23,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:23,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:23,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:23,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:23,501 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:23,501 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:23,502 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:23,502 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:23,502 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:23,502 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:23,503 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:23,503 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:23,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:23,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:23,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:23,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:23,503 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:23,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:23,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:23,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:23,504 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:23,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:23,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:23,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:23,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:23,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:23,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:23,505 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:23,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:23,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:23,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:23,507 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:23,507 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:23,507 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:23,507 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:23,507 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:23,507 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:23,507 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:23,507 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:23,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:23,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:23,507 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:23,508 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:23,508 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:23,507 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:23,508 DEBUG [Thread-2269] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:23,508 DEBUG [Thread-2269] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:23,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:23,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:23,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(9158): Flush requested on 5067d98ca5f8d3760c5d0eb2b8bb06dc 2023-04-19 20:55:23,535 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 5067d98ca5f8d3760c5d0eb2b8bb06dc 3/3 column families, dataSize=140.89 KB heapSize=405.61 KB 2023-04-19 20:55:23,535 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5067d98ca5f8d3760c5d0eb2b8bb06dc, store=A 2023-04-19 20:55:23,535 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-19 20:55:23,535 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5067d98ca5f8d3760c5d0eb2b8bb06dc, store=B 2023-04-19 20:55:23,535 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-19 20:55:23,535 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5067d98ca5f8d3760c5d0eb2b8bb06dc, store=C 2023-04-19 20:55:23,535 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-19 20:55:23,555 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:23,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56676 deadline: 1681937783550, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:23,562 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:23,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56654 deadline: 1681937783553, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:23,562 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:23,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56670 deadline: 1681937783554, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:23,608 DEBUG [Thread-2269] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:23,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:23,610 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:23,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:23,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:23,611 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:23,611 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:23,611 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:23,611 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:23,611 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:23,612 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:23,613 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:23,613 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:23,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:23,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:23,613 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:23,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:23,613 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:23,613 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:23,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:23,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:23,614 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:23,614 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:55:23,614 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:23,614 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:23,614 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:23,614 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:23,614 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:23,614 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:23,614 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:23,614 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. started... 2023-04-19 20:55:23,614 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. as already flushing 2023-04-19 20:55:23,614 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:23,614 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:23,615 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:23,615 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:23,615 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:23,615 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:23,617 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:23,617 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:23,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:23,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:23,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:23,617 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:23,617 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:23,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:23,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:23,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:23,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:23,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:23,618 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:23,618 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@139f4fe4[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:23,618 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:23,618 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:23,619 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:23,619 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:23,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:23,620 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:23,620 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:23,620 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:23,620 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:23,620 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:23,630 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:23,630 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:23,630 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:23,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:23,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:23,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:23,630 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:23,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:23,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:23,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:23,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:23,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:23,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:23,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:23,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:23,632 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:23,632 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:23,632 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:23,632 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:23,632 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:23,632 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:23,634 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:23,634 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:23,634 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:23,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:23,634 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:23,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:23,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:23,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:23,634 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:23,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:23,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:23,635 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:23,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:23,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:23,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:23,636 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:23,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:23,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:23,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:23,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:23,637 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:23,638 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:23,638 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:23,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:23,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:23,638 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:23,638 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:23,638 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:23,638 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:23,638 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:23,638 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:23,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:23,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:23,638 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:23,638 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:23,638 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:23,638 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:55:23,638 DEBUG [Thread-2269] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:23,638 DEBUG [Thread-2269] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:23,664 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:23,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56676 deadline: 1681937783656, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:23,665 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:23,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56654 deadline: 1681937783663, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:23,665 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:23,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56670 deadline: 1681937783663, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:23,738 DEBUG [Thread-2269] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:23,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:23,740 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:23,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:23,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:23,742 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:23,742 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:23,742 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:23,742 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:23,742 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:23,742 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:23,743 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:23,743 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:23,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:23,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:23,744 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:23,744 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:23,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:23,744 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:23,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:23,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:23,745 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:23,745 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:55:23,745 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:23,745 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:23,745 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:23,745 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:23,745 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:23,745 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:23,745 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:23,745 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. started... 2023-04-19 20:55:23,745 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. as already flushing 2023-04-19 20:55:23,746 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:23,746 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:23,746 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:23,746 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:23,746 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:23,746 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:23,749 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:23,749 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:23,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:23,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:23,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:23,749 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:23,749 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:23,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:23,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:23,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:23,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:23,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:23,752 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:23,752 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:23,752 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@57fe29d9[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:23,752 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:23,753 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:23,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:23,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:23,753 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:23,753 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:23,753 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:23,753 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:23,753 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:23,754 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:23,754 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:23,754 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:23,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:23,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:23,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:23,755 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:23,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:23,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:23,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:23,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:23,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:23,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:23,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:23,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:23,758 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:23,758 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:23,758 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:23,758 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:23,758 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:23,758 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:23,760 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:23,760 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:23,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:23,760 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:23,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:23,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:23,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:23,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:23,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:23,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:23,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:23,761 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:23,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:23,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:23,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:23,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:23,762 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:23,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:23,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:23,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:23,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:23,765 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:23,765 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:23,765 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:23,765 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:23,765 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:23,765 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:23,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:23,765 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:23,765 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:23,765 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:23,766 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:23,766 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:23,766 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:23,766 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:23,766 DEBUG [Thread-2269] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:23,766 DEBUG [Thread-2269] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:23,768 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:23,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:23,867 DEBUG [Thread-2269] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:23,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:23,870 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:23,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:23,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:23,871 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:23,871 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:23,871 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:23,871 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:23,872 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:23,872 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:23,872 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:23,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56654 deadline: 1681937783867, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:23,872 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:23,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56676 deadline: 1681937783867, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:23,873 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:23,873 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:23,873 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:23,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:23,873 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:23,873 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:23,873 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:23,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56670 deadline: 1681937783867, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:23,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:23,874 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:23,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:23,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:23,875 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:23,875 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:55:23,875 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:23,875 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:23,875 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:23,875 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:23,876 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:23,876 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:23,876 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:23,876 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. started... 2023-04-19 20:55:23,876 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. as already flushing 2023-04-19 20:55:23,876 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:23,876 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:23,876 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:23,876 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:23,876 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:23,876 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:23,879 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:23,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:23,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:23,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:23,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:23,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:23,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:23,881 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:23,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:23,881 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:23,881 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:23,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:23,881 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:23,881 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ab9402e[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:23,881 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:23,882 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:23,883 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:23,883 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:23,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:23,883 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:23,883 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:23,883 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:23,883 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:23,883 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:23,885 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:23,885 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:23,885 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:23,887 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:23,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:23,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:23,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:23,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:23,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:23,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:23,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:23,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:23,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:23,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:23,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:23,890 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:23,890 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:23,890 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:23,890 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:23,890 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:23,890 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:23,892 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:23,892 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:23,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:23,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:23,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:23,892 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:23,892 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:23,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:23,892 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:23,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:23,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:23,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:23,894 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:23,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:23,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:23,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:23,895 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:23,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:23,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:23,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:23,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:23,897 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:23,897 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:23,897 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:23,897 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:23,899 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:23,899 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:23,899 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:23,899 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:23,899 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:23,899 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:23,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:23,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:23,899 DEBUG [Thread-2269] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:23,899 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:23,899 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:23,899 DEBUG [Thread-2269] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:23,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:23,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:23,953 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230419484d94f6ed974ad9a921ea43a9f467f3 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419484d94f6ed974ad9a921ea43a9f467f3 2023-04-19 20:55:23,966 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=244, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp/A/c2e0e314b81742009f61561aade9f5c4 2023-04-19 20:55:23,999 DEBUG [Thread-2269] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:24,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:24,002 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:24,003 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=244 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp/B/b0f6c9b9d2b04097bb7cfd9eac915eb8 2023-04-19 20:55:24,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:24,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:24,007 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:24,007 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:24,008 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:24,008 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:24,008 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:24,008 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:24,009 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:24,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:24,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:24,010 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:24,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:24,010 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:24,011 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:24,011 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:24,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:24,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:24,012 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:24,020 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:55:24,020 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:24,020 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:24,020 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:24,020 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:24,020 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:24,020 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:24,020 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:24,020 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. started... 2023-04-19 20:55:24,021 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. as already flushing 2023-04-19 20:55:24,021 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:24,021 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:24,021 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:24,021 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:24,021 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:24,021 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:24,023 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:24,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:24,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:24,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:24,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:24,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:24,024 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:24,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:24,025 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:24,025 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:24,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:24,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:24,025 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:24,025 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@58502249[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:24,025 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:24,025 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:24,027 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:24,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:24,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:24,027 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:24,027 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:24,027 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:24,027 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:24,027 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:24,030 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:24,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:24,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:24,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:24,030 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:24,030 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:24,031 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:24,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:24,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:24,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:24,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:24,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:24,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:24,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:24,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:24,036 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:24,036 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:24,036 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:24,036 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:24,036 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:24,036 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:24,038 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=244 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp/C/2bd2594f12a24cb2a90bc65be23be928 2023-04-19 20:55:24,042 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:24,042 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:24,042 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:24,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:24,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:24,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:24,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:24,045 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:24,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:24,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:24,045 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:24,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:24,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:24,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:24,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:24,048 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:24,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:24,053 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:24,053 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:24,053 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:24,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:24,053 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:24,054 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:24,054 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:24,054 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:24,054 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:24,054 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:24,053 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:24,054 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:24,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:24,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:24,054 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:24,054 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:24,054 DEBUG [Thread-2269] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:24,054 DEBUG [Thread-2269] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:24,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:24,057 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:55:24,058 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp/A/c2e0e314b81742009f61561aade9f5c4 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/c2e0e314b81742009f61561aade9f5c4 2023-04-19 20:55:24,063 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/c2e0e314b81742009f61561aade9f5c4, entries=200, sequenceid=244, filesize=32.1 K 2023-04-19 20:55:24,063 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp/B/b0f6c9b9d2b04097bb7cfd9eac915eb8 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/b0f6c9b9d2b04097bb7cfd9eac915eb8 2023-04-19 20:55:24,067 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/b0f6c9b9d2b04097bb7cfd9eac915eb8, entries=150, sequenceid=244, filesize=11.8 K 2023-04-19 20:55:24,069 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp/C/2bd2594f12a24cb2a90bc65be23be928 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/2bd2594f12a24cb2a90bc65be23be928 2023-04-19 20:55:24,073 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/2bd2594f12a24cb2a90bc65be23be928, entries=150, sequenceid=244, filesize=11.8 K 2023-04-19 20:55:24,074 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~405.56 KB/415296, currentSize=40.25 KB/41220 for 5067d98ca5f8d3760c5d0eb2b8bb06dc in 539ms, sequenceid=244, compaction requested=true 2023-04-19 20:55:24,074 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 5067d98ca5f8d3760c5d0eb2b8bb06dc: 2023-04-19 20:55:24,074 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 20:55:24,074 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 20:55:24,074 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-19 20:55:24,074 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 20:55:24,074 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-19 20:55:24,076 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 85054 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-19 20:55:24,076 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1912): 5067d98ca5f8d3760c5d0eb2b8bb06dc/A is initiating minor compaction (all files) 2023-04-19 20:55:24,076 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 5067d98ca5f8d3760c5d0eb2b8bb06dc/A in TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:24,076 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/3e5713ebc792425caf9af7d5349a2857, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/ce774494e7894fbab682540dbcbb5957, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/c2e0e314b81742009f61561aade9f5c4] into tmpdir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp, totalSize=83.1 K 2023-04-19 20:55:24,077 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36484 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-19 20:55:24,077 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1912): 5067d98ca5f8d3760c5d0eb2b8bb06dc/B is initiating minor compaction (all files) 2023-04-19 20:55:24,077 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5067d98ca5f8d3760c5d0eb2b8bb06dc/B in TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:24,077 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/1fabe96869f7467aa6c4c29409fb6345, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/af1bd3ce6e9345748004cbb754a4efe6, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/b0f6c9b9d2b04097bb7cfd9eac915eb8] into tmpdir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp, totalSize=35.6 K 2023-04-19 20:55:24,077 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 3e5713ebc792425caf9af7d5349a2857, keycount=150, bloomtype=ROW, size=25.7 K, encoding=NONE, compression=NONE, seqNum=193, earliestPutTs=1681937720674 2023-04-19 20:55:24,078 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting ce774494e7894fbab682540dbcbb5957, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=203, earliestPutTs=1681937721271 2023-04-19 20:55:24,078 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting c2e0e314b81742009f61561aade9f5c4, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=244, earliestPutTs=1681937722886 2023-04-19 20:55:24,078 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 1fabe96869f7467aa6c4c29409fb6345, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=193, earliestPutTs=1681937720674 2023-04-19 20:55:24,079 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting af1bd3ce6e9345748004cbb754a4efe6, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=203, earliestPutTs=1681937721271 2023-04-19 20:55:24,085 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting b0f6c9b9d2b04097bb7cfd9eac915eb8, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=244, earliestPutTs=1681937722886 2023-04-19 20:55:24,101 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] throttle.PressureAwareThroughputController(145): 5067d98ca5f8d3760c5d0eb2b8bb06dc#B#compaction#339 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 57.69 MB/second 2023-04-19 20:55:24,108 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] throttle.PressureAwareThroughputController(145): 5067d98ca5f8d3760c5d0eb2b8bb06dc#A#compaction#338 average throughput is 1.97 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 57.69 MB/second 2023-04-19 20:55:24,154 DEBUG [Thread-2269] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:24,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:24,156 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:24,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:24,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:24,160 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:24,160 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:24,160 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:24,162 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:24,163 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:24,163 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:24,164 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:24,164 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:24,164 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:24,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:24,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:24,165 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:24,165 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:24,165 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:24,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:24,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:24,166 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:24,168 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:55:24,168 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:24,168 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:24,168 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:24,168 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:24,174 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:24,174 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:24,174 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:24,174 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. started... 2023-04-19 20:55:24,174 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5067d98ca5f8d3760c5d0eb2b8bb06dc 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-19 20:55:24,174 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5067d98ca5f8d3760c5d0eb2b8bb06dc, store=A 2023-04-19 20:55:24,174 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:55:24,174 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5067d98ca5f8d3760c5d0eb2b8bb06dc, store=B 2023-04-19 20:55:24,174 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:55:24,174 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5067d98ca5f8d3760c5d0eb2b8bb06dc, store=C 2023-04-19 20:55:24,174 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:55:24,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(9158): Flush requested on 5067d98ca5f8d3760c5d0eb2b8bb06dc 2023-04-19 20:55:24,178 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. as already flushing 2023-04-19 20:55:24,189 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230419825ebeb9a334448f831c4649acd54178 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419825ebeb9a334448f831c4649acd54178 2023-04-19 20:55:24,241 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1681937724218, max timestamp=1681937724239Num uniques 859; 2023-04-19 20:55:24,241 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1681937724218, max timestamp=1681937724239Num uniques 859; 2023-04-19 20:55:24,241 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-19 20:55:24,241 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-19 20:55:24,243 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1681937724218, max timestamp=1681937724239Num uniques 859; 2023-04-19 20:55:24,243 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-19 20:55:24,314 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:24,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56670 deadline: 1681937784297, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:24,314 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:24,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56676 deadline: 1681937784298, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:24,315 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:24,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56654 deadline: 1681937784298, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:24,423 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:24,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56670 deadline: 1681937784415, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:24,424 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:24,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56676 deadline: 1681937784415, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:24,424 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:24,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56654 deadline: 1681937784416, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:24,521 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp/B/e97ade3ae4cd43618db723d1870ca093 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/e97ade3ae4cd43618db723d1870ca093 2023-04-19 20:55:24,525 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5067d98ca5f8d3760c5d0eb2b8bb06dc/B of 5067d98ca5f8d3760c5d0eb2b8bb06dc into e97ade3ae4cd43618db723d1870ca093(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 20:55:24,525 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5067d98ca5f8d3760c5d0eb2b8bb06dc: 2023-04-19 20:55:24,525 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc., storeName=5067d98ca5f8d3760c5d0eb2b8bb06dc/B, priority=13, startTime=1681937724074; duration=0sec 2023-04-19 20:55:24,525 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 20:55:24,525 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-19 20:55:24,526 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36484 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-19 20:55:24,526 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1912): 5067d98ca5f8d3760c5d0eb2b8bb06dc/C is initiating minor compaction (all files) 2023-04-19 20:55:24,526 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5067d98ca5f8d3760c5d0eb2b8bb06dc/C in TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:24,526 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/8ec186b6992642ffaafc818130f98183, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/68037ac8c97b40a4be643b017208beeb, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/2bd2594f12a24cb2a90bc65be23be928] into tmpdir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp, totalSize=35.6 K 2023-04-19 20:55:24,527 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 8ec186b6992642ffaafc818130f98183, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=193, earliestPutTs=1681937720674 2023-04-19 20:55:24,527 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 68037ac8c97b40a4be643b017208beeb, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=203, earliestPutTs=1681937721271 2023-04-19 20:55:24,527 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 2bd2594f12a24cb2a90bc65be23be928, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=244, earliestPutTs=1681937722886 2023-04-19 20:55:24,532 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] throttle.PressureAwareThroughputController(145): 5067d98ca5f8d3760c5d0eb2b8bb06dc#C#compaction#341 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 57.69 MB/second 2023-04-19 20:55:24,563 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp/A/c0d0d3c6ca2a463e9091e8ea9a542b4f as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/c0d0d3c6ca2a463e9091e8ea9a542b4f 2023-04-19 20:55:24,567 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5067d98ca5f8d3760c5d0eb2b8bb06dc/A of 5067d98ca5f8d3760c5d0eb2b8bb06dc into c0d0d3c6ca2a463e9091e8ea9a542b4f(size=25.8 K), total size for store is 25.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 20:55:24,567 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5067d98ca5f8d3760c5d0eb2b8bb06dc: 2023-04-19 20:55:24,567 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc., storeName=5067d98ca5f8d3760c5d0eb2b8bb06dc/A, priority=13, startTime=1681937724074; duration=0sec 2023-04-19 20:55:24,567 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 20:55:24,600 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=253, memsize=13.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp/A/39089e48a7004a539f5f6177094991ef 2023-04-19 20:55:24,609 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=253 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp/B/b6c293283b58443fb4d9d7f739c6605c 2023-04-19 20:55:24,619 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=253 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp/C/837e73bff1614b42b3e583dd9d6c6d0a 2023-04-19 20:55:24,623 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp/A/39089e48a7004a539f5f6177094991ef as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/39089e48a7004a539f5f6177094991ef 2023-04-19 20:55:24,630 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:24,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56676 deadline: 1681937784625, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:24,630 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:24,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56670 deadline: 1681937784625, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:24,631 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:24,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56654 deadline: 1681937784626, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:24,632 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/39089e48a7004a539f5f6177094991ef, entries=150, sequenceid=253, filesize=25.3 K 2023-04-19 20:55:24,633 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp/B/b6c293283b58443fb4d9d7f739c6605c as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/b6c293283b58443fb4d9d7f739c6605c 2023-04-19 20:55:24,636 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/b6c293283b58443fb4d9d7f739c6605c, entries=150, sequenceid=253, filesize=11.8 K 2023-04-19 20:55:24,637 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp/C/837e73bff1614b42b3e583dd9d6c6d0a as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/837e73bff1614b42b3e583dd9d6c6d0a 2023-04-19 20:55:24,640 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/837e73bff1614b42b3e583dd9d6c6d0a, entries=150, sequenceid=253, filesize=11.8 K 2023-04-19 20:55:24,641 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=147.60 KB/151140 for 5067d98ca5f8d3760c5d0eb2b8bb06dc in 467ms, sequenceid=253, compaction requested=false 2023-04-19 20:55:24,641 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5067d98ca5f8d3760c5d0eb2b8bb06dc: 2023-04-19 20:55:24,641 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:24,641 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:24,641 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:24,641 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:24,641 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:24,642 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:24,643 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:24,643 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:24,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:24,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:24,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:24,643 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:24,643 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:24,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:24,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:24,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:24,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:24,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:24,645 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:24,645 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3159a6c8[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:24,645 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:24,645 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:24,647 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:24,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:24,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:24,647 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:24,647 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:24,647 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:24,647 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:24,648 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:24,649 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:24,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:24,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:24,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:24,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:24,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:24,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:24,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:24,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:24,650 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:24,650 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:24,650 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:24,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:24,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:24,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:24,651 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:24,651 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:24,651 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:24,651 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:24,651 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:24,651 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:24,653 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:24,653 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:24,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:24,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:24,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:24,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:24,653 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:24,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:24,653 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:24,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:24,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:24,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:24,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:24,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:24,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:24,654 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:24,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:24,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:24,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:24,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:24,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:24,656 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:24,656 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:24,656 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:24,656 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:24,657 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:24,656 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:24,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:24,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:24,656 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:24,657 DEBUG [Thread-2269] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:24,657 DEBUG [Thread-2269] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:24,656 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:24,657 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:55:24,657 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:24,657 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:24,657 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:24,657 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:24,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:24,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:24,757 DEBUG [Thread-2269] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:24,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:24,760 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:24,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:24,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:24,761 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:24,761 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:24,761 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:24,761 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:24,762 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:24,762 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:24,764 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:24,764 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:24,764 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:24,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:24,765 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:24,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:24,765 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:24,765 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:24,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:24,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:24,765 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:24,766 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:55:24,766 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:24,766 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:24,766 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:24,766 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:24,766 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:24,766 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:24,766 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:24,766 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. started... 2023-04-19 20:55:24,766 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5067d98ca5f8d3760c5d0eb2b8bb06dc 3/3 column families, dataSize=147.60 KB heapSize=410.77 KB 2023-04-19 20:55:24,766 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5067d98ca5f8d3760c5d0eb2b8bb06dc, store=A 2023-04-19 20:55:24,767 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 20:55:24,767 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5067d98ca5f8d3760c5d0eb2b8bb06dc, store=B 2023-04-19 20:55:24,767 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 20:55:24,767 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5067d98ca5f8d3760c5d0eb2b8bb06dc, store=C 2023-04-19 20:55:24,767 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 20:55:24,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(9158): Flush requested on 5067d98ca5f8d3760c5d0eb2b8bb06dc 2023-04-19 20:55:24,936 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. as already flushing 2023-04-19 20:55:24,941 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp/C/4d1c174328d4411db83ab3f03b088421 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/4d1c174328d4411db83ab3f03b088421 2023-04-19 20:55:24,949 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5067d98ca5f8d3760c5d0eb2b8bb06dc/C of 5067d98ca5f8d3760c5d0eb2b8bb06dc into 4d1c174328d4411db83ab3f03b088421(size=12.2 K), total size for store is 24.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 20:55:24,949 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5067d98ca5f8d3760c5d0eb2b8bb06dc: 2023-04-19 20:55:24,949 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc., storeName=5067d98ca5f8d3760c5d0eb2b8bb06dc/C, priority=13, startTime=1681937724074; duration=0sec 2023-04-19 20:55:24,949 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 20:55:24,958 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:24,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56676 deadline: 1681937784948, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:24,959 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:24,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56654 deadline: 1681937784949, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:24,964 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:24,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56670 deadline: 1681937784953, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:25,069 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:25,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56676 deadline: 1681937785059, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:25,069 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:25,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56654 deadline: 1681937785060, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:25,070 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:25,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56670 deadline: 1681937785066, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:25,180 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304198966459f2a984c91859c4a203290cac4 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304198966459f2a984c91859c4a203290cac4 2023-04-19 20:55:25,274 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:25,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56676 deadline: 1681937785270, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:25,275 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:25,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56654 deadline: 1681937785270, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:25,275 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:25,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56670 deadline: 1681937785272, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:25,454 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:25,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56626 deadline: 1681937785450, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:25,456 DEBUG [Thread-2265] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4157 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc., hostname=jenkins-hbase4.apache.org,35247,1681937616971, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-19 20:55:25,475 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:25,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56610 deadline: 1681937785471, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:25,477 DEBUG [Thread-2267] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4170 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc., hostname=jenkins-hbase4.apache.org,35247,1681937616971, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-19 20:55:25,583 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:25,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56654 deadline: 1681937785578, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:25,584 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:25,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56670 deadline: 1681937785578, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:25,584 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:25,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56676 deadline: 1681937785578, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:25,593 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=291, memsize=49.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp/A/5586d5e2347a45ee9e8ca97d97422563 2023-04-19 20:55:26,005 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=291 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp/B/8d37b8d547204ce9adb826423c855671 2023-04-19 20:55:26,089 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:26,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56654 deadline: 1681937786085, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:26,089 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:26,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56676 deadline: 1681937786086, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:26,092 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:26,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56670 deadline: 1681937786088, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:26,416 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=291 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp/C/dd9ebbf18d3d4a68a1b87918fcd26765 2023-04-19 20:55:26,420 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp/A/5586d5e2347a45ee9e8ca97d97422563 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/5586d5e2347a45ee9e8ca97d97422563 2023-04-19 20:55:26,423 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/5586d5e2347a45ee9e8ca97d97422563, entries=150, sequenceid=291, filesize=25.4 K 2023-04-19 20:55:26,424 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp/B/8d37b8d547204ce9adb826423c855671 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/8d37b8d547204ce9adb826423c855671 2023-04-19 20:55:26,427 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/8d37b8d547204ce9adb826423c855671, entries=150, sequenceid=291, filesize=11.9 K 2023-04-19 20:55:26,427 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp/C/dd9ebbf18d3d4a68a1b87918fcd26765 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/dd9ebbf18d3d4a68a1b87918fcd26765 2023-04-19 20:55:26,431 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/dd9ebbf18d3d4a68a1b87918fcd26765, entries=150, sequenceid=291, filesize=11.9 K 2023-04-19 20:55:26,431 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~410.72 KB/420576, currentSize=40.25 KB/41220 for 5067d98ca5f8d3760c5d0eb2b8bb06dc in 1665ms, sequenceid=291, compaction requested=true 2023-04-19 20:55:26,431 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5067d98ca5f8d3760c5d0eb2b8bb06dc: 2023-04-19 20:55:26,432 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:26,432 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:26,432 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:26,432 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:26,432 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:26,432 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:26,433 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:26,433 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:26,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:26,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:26,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:26,434 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:26,434 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:26,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:26,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:26,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:26,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:26,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:26,435 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:26,435 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@174fe1db[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:26,435 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:26,435 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:26,437 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:26,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:26,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:26,437 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:26,437 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:26,437 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:26,437 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:26,437 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:26,438 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:26,438 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:26,438 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:26,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:26,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:26,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:26,439 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:26,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:26,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:26,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:26,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:26,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:26,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:26,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:26,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:26,441 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:26,441 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:26,441 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:26,441 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:26,441 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:26,441 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:26,443 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:26,443 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:26,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:26,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:26,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:26,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:26,443 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:26,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:26,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:26,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:26,444 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:26,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:26,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:26,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:26,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:26,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:26,445 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:26,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:26,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:26,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:26,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:26,448 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:26,448 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:26,448 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:26,448 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:26,448 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:26,448 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:26,448 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:26,448 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:26,448 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:26,448 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:26,448 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:55:26,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:26,448 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:26,448 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:26,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:26,448 DEBUG [Thread-2269] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:26,448 DEBUG [Thread-2269] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:26,449 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:26,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:26,548 DEBUG [Thread-2269] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:26,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:26,551 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:26,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:26,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:26,553 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:26,553 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:26,553 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:26,553 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:26,553 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:26,553 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:26,555 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:26,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:26,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:26,556 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:26,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:26,556 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:26,556 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:26,556 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:26,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:26,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:26,557 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:26,557 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:55:26,557 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:26,557 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:26,557 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:26,557 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:26,557 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:26,557 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:26,557 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:26,557 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. started... 2023-04-19 20:55:26,557 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5067d98ca5f8d3760c5d0eb2b8bb06dc 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-19 20:55:26,558 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5067d98ca5f8d3760c5d0eb2b8bb06dc, store=A 2023-04-19 20:55:26,558 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:55:26,558 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5067d98ca5f8d3760c5d0eb2b8bb06dc, store=B 2023-04-19 20:55:26,558 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:55:26,558 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5067d98ca5f8d3760c5d0eb2b8bb06dc, store=C 2023-04-19 20:55:26,558 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:55:26,569 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230419c232f264476c48b486bbc3a34c60e1cc to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419c232f264476c48b486bbc3a34c60e1cc 2023-04-19 20:55:26,974 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=301, memsize=13.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp/A/667c47b0a50540b7afef7896f3bc1e42 2023-04-19 20:55:27,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(9158): Flush requested on 5067d98ca5f8d3760c5d0eb2b8bb06dc 2023-04-19 20:55:27,098 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. as already flushing 2023-04-19 20:55:27,147 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681937727134, max timestamp=1681937727145Num uniques 859; 2023-04-19 20:55:27,147 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 20:55:27,149 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681937727134, max timestamp=1681937727145Num uniques 859; 2023-04-19 20:55:27,149 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 20:55:27,149 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681937727134, max timestamp=1681937727145Num uniques 859; 2023-04-19 20:55:27,149 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 20:55:27,206 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1681937727159, max timestamp=1681937727203Num uniques 1018; 2023-04-19 20:55:27,207 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1681937727159, max timestamp=1681937727203Num uniques 1018; 2023-04-19 20:55:27,207 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=12268, suffix heap size=133840,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=268 2023-04-19 20:55:27,207 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=12268, suffix heap size=133840,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=268 2023-04-19 20:55:27,207 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1681937727159, max timestamp=1681937727203Num uniques 1018; 2023-04-19 20:55:27,207 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=12268, suffix heap size=133840,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=268 2023-04-19 20:55:27,260 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:27,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56654 deadline: 1681937787248, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:27,268 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:27,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56670 deadline: 1681937787257, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:27,269 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:27,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56676 deadline: 1681937787258, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:27,366 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:27,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56654 deadline: 1681937787361, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:27,379 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:27,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56676 deadline: 1681937787370, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:27,379 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:27,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56670 deadline: 1681937787371, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:27,383 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=301 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp/B/cb93da1182c9466498c985a1e5403163 2023-04-19 20:55:27,578 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:27,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56654 deadline: 1681937787567, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:27,589 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:27,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56670 deadline: 1681937787581, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:27,589 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:27,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56676 deadline: 1681937787581, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:27,821 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=301 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp/C/514bc54edae5475496f0bc331b053218 2023-04-19 20:55:27,825 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp/A/667c47b0a50540b7afef7896f3bc1e42 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/667c47b0a50540b7afef7896f3bc1e42 2023-04-19 20:55:27,828 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/667c47b0a50540b7afef7896f3bc1e42, entries=150, sequenceid=301, filesize=25.4 K 2023-04-19 20:55:27,829 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp/B/cb93da1182c9466498c985a1e5403163 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/cb93da1182c9466498c985a1e5403163 2023-04-19 20:55:27,832 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/cb93da1182c9466498c985a1e5403163, entries=150, sequenceid=301, filesize=11.9 K 2023-04-19 20:55:27,832 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp/C/514bc54edae5475496f0bc331b053218 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/514bc54edae5475496f0bc331b053218 2023-04-19 20:55:27,835 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/514bc54edae5475496f0bc331b053218, entries=150, sequenceid=301, filesize=11.9 K 2023-04-19 20:55:27,836 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=140.89 KB/144270 for 5067d98ca5f8d3760c5d0eb2b8bb06dc in 1279ms, sequenceid=301, compaction requested=true 2023-04-19 20:55:27,836 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5067d98ca5f8d3760c5d0eb2b8bb06dc: 2023-04-19 20:55:27,836 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:27,836 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:27,836 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:27,836 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:27,836 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:27,836 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:27,838 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:27,838 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:27,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:27,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:27,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:27,838 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:27,838 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:27,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:27,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:27,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:27,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:27,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:27,840 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:27,840 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@27c76b2c[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:27,840 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:27,840 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:27,842 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:27,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:27,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:27,842 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:27,842 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:27,842 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:27,842 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:27,842 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:27,843 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:27,843 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:27,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:27,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:27,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:27,844 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:27,844 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:27,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:27,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:27,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:27,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:27,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:27,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:27,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:27,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:27,846 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:27,846 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:27,846 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:27,846 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:27,846 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:27,846 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:27,847 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:27,847 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:27,847 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:27,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:27,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:27,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:27,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:27,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:27,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:27,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:27,848 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:27,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:27,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:27,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:27,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:27,851 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:27,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:27,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:27,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:27,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:27,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:27,854 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:27,854 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:27,854 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:27,854 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:27,854 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:27,854 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:27,854 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:27,854 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:27,854 DEBUG [Thread-2269] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:27,854 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:27,854 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:55:27,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:27,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:27,854 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:27,854 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:27,854 DEBUG [Thread-2269] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:27,854 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:27,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:27,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:27,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(9158): Flush requested on 5067d98ca5f8d3760c5d0eb2b8bb06dc 2023-04-19 20:55:27,887 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 5067d98ca5f8d3760c5d0eb2b8bb06dc 3/3 column families, dataSize=147.60 KB heapSize=414.61 KB 2023-04-19 20:55:27,887 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5067d98ca5f8d3760c5d0eb2b8bb06dc, store=A 2023-04-19 20:55:27,888 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 20:55:27,888 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5067d98ca5f8d3760c5d0eb2b8bb06dc, store=B 2023-04-19 20:55:27,888 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 20:55:27,888 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5067d98ca5f8d3760c5d0eb2b8bb06dc, store=C 2023-04-19 20:55:27,888 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 20:55:27,910 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:27,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56654 deadline: 1681937787902, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:27,914 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230419b71ef916b00c46028e9f9db7eebb90a3 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419b71ef916b00c46028e9f9db7eebb90a3 2023-04-19 20:55:27,924 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=355, memsize=49.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp/A/0c5836c6cce74b26a94c5ea844769ba5 2023-04-19 20:55:27,924 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:27,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56676 deadline: 1681937787909, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:27,925 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:27,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56670 deadline: 1681937787910, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:27,955 DEBUG [Thread-2269] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:27,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:27,956 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:27,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:27,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:27,958 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:27,958 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:27,958 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:27,958 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:27,958 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:27,958 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:27,960 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:27,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:27,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:27,960 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:27,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:27,960 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:27,961 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:27,961 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:27,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:27,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:27,961 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:27,961 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:55:27,961 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:27,961 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:27,961 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:27,961 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:27,961 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:27,961 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:27,961 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:27,961 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. started... 2023-04-19 20:55:27,961 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. as already flushing 2023-04-19 20:55:27,962 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:27,962 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:27,962 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:27,962 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:27,962 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:27,962 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:27,963 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:27,963 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:27,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:27,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:27,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:27,963 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:27,963 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:27,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:27,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:27,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:27,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:27,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:27,965 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:27,965 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7e5c47f7[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:27,965 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:27,965 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:27,966 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:27,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:27,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:27,966 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:27,966 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:27,966 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:27,966 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:27,966 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:27,969 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:27,969 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:27,969 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:27,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:27,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:27,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:27,969 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:27,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:27,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:27,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:27,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:27,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:27,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:27,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:27,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:27,971 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:27,971 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:27,971 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:27,971 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:27,971 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:27,971 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:27,972 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:27,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:27,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:27,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:27,972 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:27,973 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:27,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:27,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:27,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:27,973 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:27,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:27,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:27,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:27,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:27,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:27,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:27,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:27,974 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:27,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:27,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:27,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:27,976 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:27,976 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:27,976 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:27,976 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:27,976 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:27,976 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:27,976 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:27,976 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:27,976 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:27,976 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:27,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:27,976 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:27,977 DEBUG [Thread-2269] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:27,977 DEBUG [Thread-2269] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:27,977 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:27,977 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:27,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:27,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:28,024 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:28,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56654 deadline: 1681937788012, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:28,038 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:28,038 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:28,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56676 deadline: 1681937788025, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:28,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56670 deadline: 1681937788026, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:28,077 DEBUG [Thread-2269] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:28,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:28,079 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:28,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:28,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:28,081 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:28,081 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:28,081 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:28,081 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:28,081 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:28,081 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:28,082 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:28,082 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:28,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:28,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:28,083 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:28,083 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:28,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:28,083 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:28,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:28,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:28,084 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:28,084 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:55:28,084 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:28,084 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:28,084 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:28,084 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:28,084 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:28,084 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:28,085 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:28,085 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. started... 2023-04-19 20:55:28,085 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. as already flushing 2023-04-19 20:55:28,085 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:28,085 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:28,085 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:28,085 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:28,085 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:28,085 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:28,087 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:28,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:28,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:28,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:28,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:28,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:28,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:28,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:28,087 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:28,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:28,089 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:28,089 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:28,089 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:28,089 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@24fd112[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:28,089 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:28,089 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:28,091 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:28,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:28,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:28,091 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:28,091 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:28,091 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:28,091 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:28,091 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:28,093 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:28,093 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:28,093 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:28,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:28,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:28,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:28,094 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:28,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:28,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:28,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:28,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:28,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:28,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:28,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:28,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:28,097 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:28,097 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:28,097 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:28,097 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:28,097 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:28,097 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:28,098 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:28,098 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:28,098 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:28,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:28,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:28,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:28,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:28,100 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:28,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:28,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:28,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:28,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:28,101 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:28,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:28,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:28,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:28,102 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:28,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:28,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:28,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:28,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:28,105 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:28,105 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:28,105 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:28,105 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:28,105 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:28,105 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:28,105 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:28,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:28,105 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:55:28,105 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:28,105 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:28,105 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:28,106 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:28,106 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:28,106 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:28,106 DEBUG [Thread-2269] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:28,106 DEBUG [Thread-2269] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:28,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:28,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:28,206 DEBUG [Thread-2269] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:28,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:28,208 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:28,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:28,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:28,209 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:28,210 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:28,210 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:28,210 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:28,210 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:28,210 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:28,211 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:28,211 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:28,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:28,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:28,212 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:28,212 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:28,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:28,212 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:28,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:28,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:28,213 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:28,213 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:55:28,213 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:28,213 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:28,213 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:28,213 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:28,213 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:28,213 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:28,213 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:28,213 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. started... 2023-04-19 20:55:28,213 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. as already flushing 2023-04-19 20:55:28,213 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:28,213 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:28,213 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:28,213 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:28,213 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:28,214 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:28,215 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:28,215 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:28,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:28,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:28,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:28,216 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:28,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:28,216 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:28,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:28,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:28,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:28,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:28,217 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:28,217 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a66b9b9[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:28,217 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:28,217 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:28,218 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:28,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:28,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:28,219 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:28,219 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:28,219 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:28,219 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:28,219 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:28,221 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:28,221 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:28,221 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:28,221 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:28,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:28,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:28,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:28,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:28,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:28,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:28,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:28,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:28,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:28,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:28,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:28,224 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:28,224 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:28,225 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:28,225 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:28,225 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:28,225 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:28,226 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:28,227 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:28,227 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:28,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:28,227 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:28,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:28,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:28,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:28,227 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:28,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:28,227 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:28,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:28,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:28,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:28,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:28,229 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:28,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:28,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:28,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:28,233 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:28,233 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:28,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:28,233 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:28,233 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:28,233 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:28,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:28,233 DEBUG [Thread-2269] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:28,233 DEBUG [Thread-2269] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:28,233 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:28,233 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:28,233 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:28,233 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:28,233 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:28,233 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:28,234 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:28,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:28,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:28,234 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:55:28,235 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:28,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56654 deadline: 1681937788226, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:28,249 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:28,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56670 deadline: 1681937788239, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:28,249 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:28,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56676 deadline: 1681937788239, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:28,333 DEBUG [Thread-2269] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:28,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:28,336 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:28,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:28,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:28,337 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:28,337 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:28,337 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:28,337 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:28,338 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=355 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp/B/5e1d15b820cb4c7abd7c39e6d81bff29 2023-04-19 20:55:28,339 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:28,339 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:28,347 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:28,347 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:28,347 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:28,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:28,347 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:28,347 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:28,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:28,348 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:28,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:28,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:28,348 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:28,349 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:55:28,349 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:28,349 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:28,349 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:28,349 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:28,349 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:28,349 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:28,349 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:28,349 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. started... 2023-04-19 20:55:28,349 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. as already flushing 2023-04-19 20:55:28,349 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:28,349 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:28,349 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:28,349 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:28,350 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:28,350 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:28,351 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:28,351 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:28,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:28,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:28,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:28,352 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:28,352 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:28,353 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=355 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp/C/611c5cfb2cba471e96fb35643f3fc14d 2023-04-19 20:55:28,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:28,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:28,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:28,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:28,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:28,354 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:28,354 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@76e431d2[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:28,354 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:28,354 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:28,356 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:28,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:28,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:28,356 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:28,356 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:28,356 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:28,357 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:28,357 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:28,357 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:28,357 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp/A/0c5836c6cce74b26a94c5ea844769ba5 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/0c5836c6cce74b26a94c5ea844769ba5 2023-04-19 20:55:28,358 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:28,359 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:28,359 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:28,359 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:28,359 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:28,359 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:28,359 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:28,359 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:28,361 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/0c5836c6cce74b26a94c5ea844769ba5, entries=200, sequenceid=355, filesize=32.3 K 2023-04-19 20:55:28,362 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp/B/5e1d15b820cb4c7abd7c39e6d81bff29 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/5e1d15b820cb4c7abd7c39e6d81bff29 2023-04-19 20:55:28,362 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:28,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:28,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:28,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:28,362 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:28,362 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:28,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:28,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:28,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:28,363 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:28,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:28,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:28,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:28,365 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:28,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:28,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:28,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:28,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:28,367 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:28,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:28,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:28,370 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/5e1d15b820cb4c7abd7c39e6d81bff29, entries=150, sequenceid=355, filesize=11.9 K 2023-04-19 20:55:28,371 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:28,371 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:28,371 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:28,371 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:28,371 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:28,371 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:28,371 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp/C/611c5cfb2cba471e96fb35643f3fc14d as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/611c5cfb2cba471e96fb35643f3fc14d 2023-04-19 20:55:28,371 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:28,371 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:28,371 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:28,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:28,371 DEBUG [Thread-2269] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:28,371 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:28,371 DEBUG [Thread-2269] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:28,371 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:28,371 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:28,371 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:28,371 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:28,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:28,375 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/611c5cfb2cba471e96fb35643f3fc14d, entries=150, sequenceid=355, filesize=11.9 K 2023-04-19 20:55:28,376 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~414.56 KB/424512, currentSize=40.25 KB/41220 for 5067d98ca5f8d3760c5d0eb2b8bb06dc in 489ms, sequenceid=355, compaction requested=true 2023-04-19 20:55:28,376 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 5067d98ca5f8d3760c5d0eb2b8bb06dc: 2023-04-19 20:55:28,376 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 20:55:28,376 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-19 20:55:28,376 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-19 20:55:28,376 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 20:55:28,376 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 20:55:28,378 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61108 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-19 20:55:28,378 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 137448 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-19 20:55:28,378 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1912): 5067d98ca5f8d3760c5d0eb2b8bb06dc/B is initiating minor compaction (all files) 2023-04-19 20:55:28,378 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1912): 5067d98ca5f8d3760c5d0eb2b8bb06dc/A is initiating minor compaction (all files) 2023-04-19 20:55:28,378 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5067d98ca5f8d3760c5d0eb2b8bb06dc/B in TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:28,378 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 5067d98ca5f8d3760c5d0eb2b8bb06dc/A in TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:28,378 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/e97ade3ae4cd43618db723d1870ca093, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/b6c293283b58443fb4d9d7f739c6605c, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/8d37b8d547204ce9adb826423c855671, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/cb93da1182c9466498c985a1e5403163, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/5e1d15b820cb4c7abd7c39e6d81bff29] into tmpdir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp, totalSize=59.7 K 2023-04-19 20:55:28,378 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/c0d0d3c6ca2a463e9091e8ea9a542b4f, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/39089e48a7004a539f5f6177094991ef, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/5586d5e2347a45ee9e8ca97d97422563, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/667c47b0a50540b7afef7896f3bc1e42, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/0c5836c6cce74b26a94c5ea844769ba5] into tmpdir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp, totalSize=134.2 K 2023-04-19 20:55:28,378 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting e97ade3ae4cd43618db723d1870ca093, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=244, earliestPutTs=1681937722886 2023-04-19 20:55:28,379 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting c0d0d3c6ca2a463e9091e8ea9a542b4f, keycount=150, bloomtype=ROW, size=25.8 K, encoding=NONE, compression=NONE, seqNum=244, earliestPutTs=1681937722886 2023-04-19 20:55:28,379 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting b6c293283b58443fb4d9d7f739c6605c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=253, earliestPutTs=1681937723543 2023-04-19 20:55:28,379 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 39089e48a7004a539f5f6177094991ef, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=253, earliestPutTs=1681937723543 2023-04-19 20:55:28,379 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 8d37b8d547204ce9adb826423c855671, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=291, earliestPutTs=1681937724280 2023-04-19 20:55:28,379 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 5586d5e2347a45ee9e8ca97d97422563, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=291, earliestPutTs=1681937724280 2023-04-19 20:55:28,380 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting cb93da1182c9466498c985a1e5403163, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=301, earliestPutTs=1681937724946 2023-04-19 20:55:28,380 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 667c47b0a50540b7afef7896f3bc1e42, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=301, earliestPutTs=1681937724946 2023-04-19 20:55:28,380 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 5e1d15b820cb4c7abd7c39e6d81bff29, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=355, earliestPutTs=1681937727243 2023-04-19 20:55:28,380 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 0c5836c6cce74b26a94c5ea844769ba5, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=355, earliestPutTs=1681937727242 2023-04-19 20:55:28,396 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] throttle.PressureAwareThroughputController(145): 5067d98ca5f8d3760c5d0eb2b8bb06dc#B#compaction#353 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 57.69 MB/second 2023-04-19 20:55:28,400 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] throttle.PressureAwareThroughputController(145): 5067d98ca5f8d3760c5d0eb2b8bb06dc#A#compaction#354 average throughput is 3.94 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 57.69 MB/second 2023-04-19 20:55:28,471 DEBUG [Thread-2269] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:28,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:28,474 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:28,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:28,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:28,475 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:28,475 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:28,475 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:28,475 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:28,475 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:28,475 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:28,476 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:28,476 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:28,477 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:28,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:28,477 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:28,477 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:28,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:28,477 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:28,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:28,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:28,478 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:28,478 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:55:28,478 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:28,478 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:28,478 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:28,478 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:28,478 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:28,478 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:28,478 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:28,478 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. started... 2023-04-19 20:55:28,478 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5067d98ca5f8d3760c5d0eb2b8bb06dc 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-19 20:55:28,479 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5067d98ca5f8d3760c5d0eb2b8bb06dc, store=A 2023-04-19 20:55:28,479 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:55:28,479 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5067d98ca5f8d3760c5d0eb2b8bb06dc, store=B 2023-04-19 20:55:28,479 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:55:28,479 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5067d98ca5f8d3760c5d0eb2b8bb06dc, store=C 2023-04-19 20:55:28,479 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:55:28,491 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304199fdabb2747684160b915f4905b1d89f8 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304199fdabb2747684160b915f4905b1d89f8 2023-04-19 20:55:28,496 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=364, memsize=13.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp/A/64e8bbbcd36c465eb635f45f88c903ef 2023-04-19 20:55:28,504 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=364 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp/B/76d4439dacdd48548706a4feb1970175 2023-04-19 20:55:28,513 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=364 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp/C/b01ff249fa494258bb916bd2bbb7d275 2023-04-19 20:55:28,516 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp/A/64e8bbbcd36c465eb635f45f88c903ef as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/64e8bbbcd36c465eb635f45f88c903ef 2023-04-19 20:55:28,520 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/64e8bbbcd36c465eb635f45f88c903ef, entries=100, sequenceid=364, filesize=18.6 K 2023-04-19 20:55:28,520 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp/B/76d4439dacdd48548706a4feb1970175 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/76d4439dacdd48548706a4feb1970175 2023-04-19 20:55:28,524 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/76d4439dacdd48548706a4feb1970175, entries=100, sequenceid=364, filesize=9.5 K 2023-04-19 20:55:28,524 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp/C/b01ff249fa494258bb916bd2bbb7d275 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/b01ff249fa494258bb916bd2bbb7d275 2023-04-19 20:55:28,528 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/b01ff249fa494258bb916bd2bbb7d275, entries=100, sequenceid=364, filesize=9.5 K 2023-04-19 20:55:28,529 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=0 B/0 for 5067d98ca5f8d3760c5d0eb2b8bb06dc in 51ms, sequenceid=364, compaction requested=true 2023-04-19 20:55:28,529 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5067d98ca5f8d3760c5d0eb2b8bb06dc: 2023-04-19 20:55:28,529 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:28,529 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:28,529 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:28,529 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:28,529 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:28,529 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:28,530 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:28,530 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:28,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:28,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:28,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:28,531 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:28,531 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:28,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:28,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:28,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:28,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:28,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:28,532 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:28,532 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a9bb306[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:28,532 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:28,533 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:28,535 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:28,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:28,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:28,535 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:28,535 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:28,535 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:28,535 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:28,535 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:28,537 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:28,537 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:28,537 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:28,537 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:28,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:28,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:28,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:28,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:28,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:28,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:28,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:28,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:28,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:28,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:28,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:28,539 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:28,540 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:28,540 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:28,540 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:28,540 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:28,540 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:28,541 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:28,541 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:28,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:28,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:28,541 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:28,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:28,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:28,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:28,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:28,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:28,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:28,542 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:28,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:28,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:28,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:28,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:28,543 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:28,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:28,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:28,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:28,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:28,545 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:28,545 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:28,545 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:28,546 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:28,546 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:28,546 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:55:28,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:28,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:28,546 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:28,546 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:28,546 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:28,546 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:28,546 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:28,546 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:28,546 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:28,546 DEBUG [Thread-2269] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:28,546 DEBUG [Thread-2269] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:28,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:28,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:28,646 DEBUG [Thread-2269] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:28,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:28,649 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:28,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:28,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:28,650 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:28,650 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:28,650 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:28,650 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:28,650 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:28,650 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:28,653 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:28,653 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:28,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:28,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:28,653 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:28,653 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:28,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:28,653 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:28,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:28,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:28,654 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:28,654 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:28,654 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:28,654 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:28,654 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:28,654 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:28,654 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:28,654 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:28,654 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. started... 2023-04-19 20:55:28,654 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5067d98ca5f8d3760c5d0eb2b8bb06dc 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-19 20:55:28,655 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5067d98ca5f8d3760c5d0eb2b8bb06dc, store=A 2023-04-19 20:55:28,655 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:55:28,655 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5067d98ca5f8d3760c5d0eb2b8bb06dc, store=B 2023-04-19 20:55:28,655 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:55:28,655 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5067d98ca5f8d3760c5d0eb2b8bb06dc, store=C 2023-04-19 20:55:28,655 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:55:28,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(9158): Flush requested on 5067d98ca5f8d3760c5d0eb2b8bb06dc 2023-04-19 20:55:28,666 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. as already flushing 2023-04-19 20:55:28,710 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681937728697, max timestamp=1681937728708Num uniques 859; 2023-04-19 20:55:28,710 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-19 20:55:28,710 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681937728697, max timestamp=1681937728708Num uniques 859; 2023-04-19 20:55:28,710 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-19 20:55:28,710 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681937728697, max timestamp=1681937728708Num uniques 859; 2023-04-19 20:55:28,711 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-19 20:55:28,782 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1681937728768, max timestamp=1681937728780Num uniques 1068; 2023-04-19 20:55:28,782 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=9978, suffix heap size=139040,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=218 2023-04-19 20:55:28,783 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1681937728768, max timestamp=1681937728780Num uniques 1068; 2023-04-19 20:55:28,783 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=9978, suffix heap size=139040,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=218 2023-04-19 20:55:28,783 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1681937728768, max timestamp=1681937728780Num uniques 1068; 2023-04-19 20:55:28,783 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=9978, suffix heap size=139040,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=218 2023-04-19 20:55:28,806 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp/B/3df8315ed49142128a18327ccd4909ca as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/3df8315ed49142128a18327ccd4909ca 2023-04-19 20:55:28,810 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 5067d98ca5f8d3760c5d0eb2b8bb06dc/B of 5067d98ca5f8d3760c5d0eb2b8bb06dc into 3df8315ed49142128a18327ccd4909ca(size=12.5 K), total size for store is 22.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 20:55:28,810 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5067d98ca5f8d3760c5d0eb2b8bb06dc: 2023-04-19 20:55:28,810 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc., storeName=5067d98ca5f8d3760c5d0eb2b8bb06dc/B, priority=11, startTime=1681937728376; duration=0sec 2023-04-19 20:55:28,810 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 20:55:28,810 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-19 20:55:28,812 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 70850 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-19 20:55:28,812 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1912): 5067d98ca5f8d3760c5d0eb2b8bb06dc/C is initiating minor compaction (all files) 2023-04-19 20:55:28,812 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5067d98ca5f8d3760c5d0eb2b8bb06dc/C in TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:28,813 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/4d1c174328d4411db83ab3f03b088421, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/837e73bff1614b42b3e583dd9d6c6d0a, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/dd9ebbf18d3d4a68a1b87918fcd26765, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/514bc54edae5475496f0bc331b053218, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/611c5cfb2cba471e96fb35643f3fc14d, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/b01ff249fa494258bb916bd2bbb7d275] into tmpdir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp, totalSize=69.2 K 2023-04-19 20:55:28,813 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 4d1c174328d4411db83ab3f03b088421, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=244, earliestPutTs=1681937722886 2023-04-19 20:55:28,813 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 837e73bff1614b42b3e583dd9d6c6d0a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=253, earliestPutTs=1681937723543 2023-04-19 20:55:28,814 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting dd9ebbf18d3d4a68a1b87918fcd26765, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=291, earliestPutTs=1681937724280 2023-04-19 20:55:28,814 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 514bc54edae5475496f0bc331b053218, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=301, earliestPutTs=1681937724946 2023-04-19 20:55:28,814 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 611c5cfb2cba471e96fb35643f3fc14d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=355, earliestPutTs=1681937727243 2023-04-19 20:55:28,815 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting b01ff249fa494258bb916bd2bbb7d275, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=364, earliestPutTs=1681937727900 2023-04-19 20:55:28,822 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] throttle.PressureAwareThroughputController(145): 5067d98ca5f8d3760c5d0eb2b8bb06dc#C#compaction#359 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 57.69 MB/second 2023-04-19 20:55:28,840 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1681937728811, max timestamp=1681937728838Num uniques 1077; 2023-04-19 20:55:28,840 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=12682, suffix heap size=139976,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=277 2023-04-19 20:55:28,841 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1681937728811, max timestamp=1681937728838Num uniques 1077; 2023-04-19 20:55:28,841 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=12682, suffix heap size=139976,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=277 2023-04-19 20:55:28,841 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1681937728811, max timestamp=1681937728838Num uniques 1077; 2023-04-19 20:55:28,841 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=12682, suffix heap size=139976,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=277 2023-04-19 20:55:28,899 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:28,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56676 deadline: 1681937788889, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:28,900 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1681937728887, max timestamp=1681937728897Num uniques 1136; 2023-04-19 20:55:28,900 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=52026, new segment data size=8516, suffix heap size=146112,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1136, new segment cells count=186 2023-04-19 20:55:28,900 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1681937728887, max timestamp=1681937728897Num uniques 1136; 2023-04-19 20:55:28,900 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1681937728887, max timestamp=1681937728897Num uniques 1136; 2023-04-19 20:55:28,900 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=52026, new segment data size=8516, suffix heap size=146112,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1136, new segment cells count=186 2023-04-19 20:55:28,900 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=52026, new segment data size=8516, suffix heap size=146112,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1136, new segment cells count=186 2023-04-19 20:55:28,975 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1681937728945, max timestamp=1681937728973Num uniques 1045; 2023-04-19 20:55:28,975 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-04-19 20:55:28,975 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1681937728945, max timestamp=1681937728973Num uniques 1045; 2023-04-19 20:55:28,976 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-04-19 20:55:28,976 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1681937728945, max timestamp=1681937728973Num uniques 1045; 2023-04-19 20:55:28,976 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-04-19 20:55:29,037 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=254, cellSize=11630, totalHeapSize=26656, min timestamp=1681937729001, max timestamp=1681937729034Num uniques 1054; 2023-04-19 20:55:29,037 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=254, cellSize=11630, totalHeapSize=26656, min timestamp=1681937729001, max timestamp=1681937729034Num uniques 1054; 2023-04-19 20:55:29,037 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=11630, suffix heap size=137584,new segment heap size=26656 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=254 2023-04-19 20:55:29,037 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=11630, suffix heap size=137584,new segment heap size=26656 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=254 2023-04-19 20:55:29,037 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=254, cellSize=11630, totalHeapSize=26656, min timestamp=1681937729001, max timestamp=1681937729034Num uniques 1054; 2023-04-19 20:55:29,037 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=11630, suffix heap size=137584,new segment heap size=26656 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=254 2023-04-19 20:55:29,083 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304192b2fe1cc8fc341edae1d0b8db7a5f0eb to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304192b2fe1cc8fc341edae1d0b8db7a5f0eb 2023-04-19 20:55:29,124 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1681937729064, max timestamp=1681937729121Num uniques 1113; 2023-04-19 20:55:29,124 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=50968, new segment data size=9748, suffix heap size=143720,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1113, new segment cells count=213 2023-04-19 20:55:29,124 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1681937729064, max timestamp=1681937729121Num uniques 1113; 2023-04-19 20:55:29,124 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=50968, new segment data size=9748, suffix heap size=143720,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1113, new segment cells count=213 2023-04-19 20:55:29,124 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1681937729064, max timestamp=1681937729121Num uniques 1113; 2023-04-19 20:55:29,124 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=50968, new segment data size=9748, suffix heap size=143720,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1113, new segment cells count=213 2023-04-19 20:55:29,197 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:29,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56654 deadline: 1681937789196, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:29,199 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10155, totalHeapSize=23328, min timestamp=1681937729176, max timestamp=1681937729196Num uniques 1072; 2023-04-19 20:55:29,199 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=49085, new segment data size=10155, suffix heap size=139456,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=222 2023-04-19 20:55:29,199 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1681937729176, max timestamp=1681937729196Num uniques 1072; 2023-04-19 20:55:29,199 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=10162, suffix heap size=139456,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=222 2023-04-19 20:55:29,199 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10160, totalHeapSize=23328, min timestamp=1681937729176, max timestamp=1681937729196Num uniques 1072; 2023-04-19 20:55:29,199 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=49090, new segment data size=10160, suffix heap size=139456,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=222 2023-04-19 20:55:29,219 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp/A/d1a42b823b8f4dd89fb9892c654e087d as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/d1a42b823b8f4dd89fb9892c654e087d 2023-04-19 20:55:29,224 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 5067d98ca5f8d3760c5d0eb2b8bb06dc/A of 5067d98ca5f8d3760c5d0eb2b8bb06dc into d1a42b823b8f4dd89fb9892c654e087d(size=26.1 K), total size for store is 44.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 20:55:29,224 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5067d98ca5f8d3760c5d0eb2b8bb06dc: 2023-04-19 20:55:29,224 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc., storeName=5067d98ca5f8d3760c5d0eb2b8bb06dc/A, priority=11, startTime=1681937728376; duration=0sec 2023-04-19 20:55:29,224 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 20:55:29,229 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp/C/18084385e97d480d9ac6b49db46c2744 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/18084385e97d480d9ac6b49db46c2744 2023-04-19 20:55:29,244 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 5067d98ca5f8d3760c5d0eb2b8bb06dc/C of 5067d98ca5f8d3760c5d0eb2b8bb06dc into 18084385e97d480d9ac6b49db46c2744(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 20:55:29,244 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5067d98ca5f8d3760c5d0eb2b8bb06dc: 2023-04-19 20:55:29,244 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc., storeName=5067d98ca5f8d3760c5d0eb2b8bb06dc/C, priority=10, startTime=1681937728376; duration=0sec 2023-04-19 20:55:29,244 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 20:55:29,265 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1681937729227, max timestamp=1681937729261Num uniques 1081; 2023-04-19 20:55:29,265 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=10576, suffix heap size=140392,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=231 2023-04-19 20:55:29,265 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1681937729227, max timestamp=1681937729261Num uniques 1081; 2023-04-19 20:55:29,265 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1681937729227, max timestamp=1681937729261Num uniques 1081; 2023-04-19 20:55:29,265 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=10576, suffix heap size=140392,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=231 2023-04-19 20:55:29,265 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=10576, suffix heap size=140392,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=231 2023-04-19 20:55:29,338 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10990, totalHeapSize=25200, min timestamp=1681937729313, max timestamp=1681937729335Num uniques 1090; 2023-04-19 20:55:29,338 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=10990, suffix heap size=141328,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=240 2023-04-19 20:55:29,339 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10990, totalHeapSize=25200, min timestamp=1681937729313, max timestamp=1681937729335Num uniques 1090; 2023-04-19 20:55:29,339 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=10990, suffix heap size=141328,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=240 2023-04-19 20:55:29,339 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10990, totalHeapSize=25200, min timestamp=1681937729313, max timestamp=1681937729335Num uniques 1090; 2023-04-19 20:55:29,339 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=10990, suffix heap size=141328,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=240 2023-04-19 20:55:29,396 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=249, cellSize=11404, totalHeapSize=26136, min timestamp=1681937729367, max timestamp=1681937729393Num uniques 1099; 2023-04-19 20:55:29,396 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=50334, new segment data size=11404, suffix heap size=142264,new segment heap size=26136 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1099, new segment cells count=249 2023-04-19 20:55:29,397 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9114, totalHeapSize=20936, min timestamp=1681937729367, max timestamp=1681937729393Num uniques 1099; 2023-04-19 20:55:29,397 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=50334, new segment data size=9114, suffix heap size=142264,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1099, new segment cells count=199 2023-04-19 20:55:29,397 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=249, cellSize=11404, totalHeapSize=26136, min timestamp=1681937729367, max timestamp=1681937729393Num uniques 1099; 2023-04-19 20:55:29,397 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=50334, new segment data size=11404, suffix heap size=142264,new segment heap size=26136 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1099, new segment cells count=249 2023-04-19 20:55:29,452 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=208, cellSize=9520, totalHeapSize=21872, min timestamp=1681937729437, max timestamp=1681937729450Num uniques 1108; 2023-04-19 20:55:29,452 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=50740, new segment data size=9520, suffix heap size=143200,new segment heap size=21872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1108, new segment cells count=208 2023-04-19 20:55:29,453 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=208, cellSize=9520, totalHeapSize=21872, min timestamp=1681937729437, max timestamp=1681937729450Num uniques 1108; 2023-04-19 20:55:29,453 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=50740, new segment data size=9520, suffix heap size=143200,new segment heap size=21872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1108, new segment cells count=208 2023-04-19 20:55:29,453 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=208, cellSize=9520, totalHeapSize=21872, min timestamp=1681937729437, max timestamp=1681937729450Num uniques 1058; 2023-04-19 20:55:29,453 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=48450, new segment data size=9520, suffix heap size=138000,new segment heap size=21872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1058, new segment cells count=208 2023-04-19 20:55:29,487 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:29,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 253 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56670 deadline: 1681937789482, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:29,488 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=217, cellSize=9932, totalHeapSize=22808, min timestamp=1681937729472, max timestamp=1681937729485Num uniques 1067; 2023-04-19 20:55:29,488 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=217, cellSize=9932, totalHeapSize=22808, min timestamp=1681937729472, max timestamp=1681937729485Num uniques 1067; 2023-04-19 20:55:29,488 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=217, cellSize=9932, totalHeapSize=22808, min timestamp=1681937729472, max timestamp=1681937729485Num uniques 1067; 2023-04-19 20:55:29,488 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=48862, new segment data size=9932, suffix heap size=138936,new segment heap size=22808 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1067, new segment cells count=217 2023-04-19 20:55:29,488 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=48862, new segment data size=9932, suffix heap size=138936,new segment heap size=22808 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1067, new segment cells count=217 2023-04-19 20:55:29,488 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=48862, new segment data size=9932, suffix heap size=138936,new segment heap size=22808 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1067, new segment cells count=217 2023-04-19 20:55:29,490 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=371, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp/A/630b0bf052924e7a912e4e926138960b 2023-04-19 20:55:29,514 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=326, cellSize=14926, totalHeapSize=34144, min timestamp=1681937729500, max timestamp=1681937729511Num uniques 1076; 2023-04-19 20:55:29,514 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=49276, new segment data size=14926, suffix heap size=139872,new segment heap size=34144 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1076, new segment cells count=326 2023-04-19 20:55:29,514 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=326, cellSize=14926, totalHeapSize=34144, min timestamp=1681937729500, max timestamp=1681937729511Num uniques 1076; 2023-04-19 20:55:29,514 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=49276, new segment data size=14926, suffix heap size=139872,new segment heap size=34144 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1076, new segment cells count=326 2023-04-19 20:55:29,514 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=326, cellSize=14926, totalHeapSize=34144, min timestamp=1681937729500, max timestamp=1681937729511Num uniques 1076; 2023-04-19 20:55:29,514 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=49276, new segment data size=14926, suffix heap size=139872,new segment heap size=34144 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1076, new segment cells count=326 2023-04-19 20:55:29,544 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:29,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 252 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56654 deadline: 1681937789537, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:29,545 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=235, cellSize=10760, totalHeapSize=24680, min timestamp=1681937729530, max timestamp=1681937729542Num uniques 1185; 2023-04-19 20:55:29,545 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=54270, new segment data size=10760, suffix heap size=151208,new segment heap size=24680 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1185, new segment cells count=235 2023-04-19 20:55:29,545 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=235, cellSize=10760, totalHeapSize=24680, min timestamp=1681937729530, max timestamp=1681937729542Num uniques 1185; 2023-04-19 20:55:29,545 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=54270, new segment data size=10760, suffix heap size=151208,new segment heap size=24680 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1185, new segment cells count=235 2023-04-19 20:55:29,545 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=235, cellSize=10760, totalHeapSize=24680, min timestamp=1681937729530, max timestamp=1681937729542Num uniques 1185; 2023-04-19 20:55:29,545 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=54270, new segment data size=10760, suffix heap size=151208,new segment heap size=24680 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1185, new segment cells count=235 2023-04-19 20:55:29,592 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:29,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56610 deadline: 1681937789584, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:29,592 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:29,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 255 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56670 deadline: 1681937789589, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:29,592 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=294, cellSize=13464, totalHeapSize=30816, min timestamp=1681937729564, max timestamp=1681937729589Num uniques 1094; 2023-04-19 20:55:29,593 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=50104, new segment data size=13464, suffix heap size=141744,new segment heap size=30816 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1094, new segment cells count=294 2023-04-19 20:55:29,594 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=294, cellSize=13464, totalHeapSize=30816, min timestamp=1681937729564, max timestamp=1681937729589Num uniques 1094; 2023-04-19 20:55:29,594 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=50104, new segment data size=13464, suffix heap size=141744,new segment heap size=30816 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1094, new segment cells count=294 2023-04-19 20:55:29,595 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=294, cellSize=13464, totalHeapSize=30816, min timestamp=1681937729564, max timestamp=1681937729589Num uniques 1094; 2023-04-19 20:55:29,595 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=50104, new segment data size=13464, suffix heap size=141744,new segment heap size=30816 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1094, new segment cells count=294 2023-04-19 20:55:29,645 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=203, cellSize=9288, totalHeapSize=21352, min timestamp=1681937729624, max timestamp=1681937729641Num uniques 1153; 2023-04-19 20:55:29,646 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=52798, new segment data size=9288, suffix heap size=147880,new segment heap size=21352 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1153, new segment cells count=203 2023-04-19 20:55:29,646 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=203, cellSize=9288, totalHeapSize=21352, min timestamp=1681937729624, max timestamp=1681937729641Num uniques 1153; 2023-04-19 20:55:29,646 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=203, cellSize=9288, totalHeapSize=21352, min timestamp=1681937729624, max timestamp=1681937729641Num uniques 1153; 2023-04-19 20:55:29,646 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=52798, new segment data size=9288, suffix heap size=147880,new segment heap size=21352 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1153, new segment cells count=203 2023-04-19 20:55:29,646 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=52798, new segment data size=9288, suffix heap size=147880,new segment heap size=21352 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1153, new segment cells count=203 2023-04-19 20:55:29,683 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=262, cellSize=11992, totalHeapSize=27488, min timestamp=1681937729670, max timestamp=1681937729680Num uniques 1062; 2023-04-19 20:55:29,683 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=48632, new segment data size=11992, suffix heap size=138416,new segment heap size=27488 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1062, new segment cells count=262 2023-04-19 20:55:29,683 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=262, cellSize=11992, totalHeapSize=27488, min timestamp=1681937729670, max timestamp=1681937729680Num uniques 1062; 2023-04-19 20:55:29,683 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=48632, new segment data size=11992, suffix heap size=138416,new segment heap size=27488 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1062, new segment cells count=262 2023-04-19 20:55:29,683 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=262, cellSize=11992, totalHeapSize=27488, min timestamp=1681937729670, max timestamp=1681937729680Num uniques 1062; 2023-04-19 20:55:29,683 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=48632, new segment data size=11992, suffix heap size=138416,new segment heap size=27488 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1062, new segment cells count=262 2023-04-19 20:55:29,740 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:29,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 263 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56654 deadline: 1681937789734, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:29,740 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:29,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56610 deadline: 1681937789735, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:29,741 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:29,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56626 deadline: 1681937789738, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:29,741 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=221, cellSize=10116, totalHeapSize=23224, min timestamp=1681937729725, max timestamp=1681937729739Num uniques 1121; 2023-04-19 20:55:29,741 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=171, cellSize=7826, totalHeapSize=18024, min timestamp=1681937729733, max timestamp=1681937729739Num uniques 1121; 2023-04-19 20:55:29,741 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=51336, new segment data size=10116, suffix heap size=144552,new segment heap size=23224 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1121, new segment cells count=221 2023-04-19 20:55:29,741 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=51336, new segment data size=7826, suffix heap size=144552,new segment heap size=18024 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1121, new segment cells count=171 2023-04-19 20:55:29,742 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=171, cellSize=7826, totalHeapSize=18024, min timestamp=1681937729733, max timestamp=1681937729739Num uniques 1121; 2023-04-19 20:55:29,742 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=51336, new segment data size=7826, suffix heap size=144552,new segment heap size=18024 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1121, new segment cells count=171 2023-04-19 20:55:29,828 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=230, cellSize=10530, totalHeapSize=24160, min timestamp=1681937729801, max timestamp=1681937729825Num uniques 1080; 2023-04-19 20:55:29,828 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=230, cellSize=10530, totalHeapSize=24160, min timestamp=1681937729801, max timestamp=1681937729825Num uniques 1030; 2023-04-19 20:55:29,828 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=49460, new segment data size=10530, suffix heap size=140288,new segment heap size=24160 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1080, new segment cells count=230 2023-04-19 20:55:29,828 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=230, cellSize=10530, totalHeapSize=24160, min timestamp=1681937729801, max timestamp=1681937729825Num uniques 1030; 2023-04-19 20:55:29,828 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47170, new segment data size=10530, suffix heap size=135088,new segment heap size=24160 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1030, new segment cells count=230 2023-04-19 20:55:29,828 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=47170, new segment data size=10530, suffix heap size=135088,new segment heap size=24160 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1030, new segment cells count=230 2023-04-19 20:55:29,861 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=339, cellSize=15524, totalHeapSize=35496, min timestamp=1681937729851, max timestamp=1681937729859Num uniques 1089; 2023-04-19 20:55:29,861 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=49874, new segment data size=15524, suffix heap size=141224,new segment heap size=35496 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1089, new segment cells count=339 2023-04-19 20:55:29,862 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=339, cellSize=15524, totalHeapSize=35496, min timestamp=1681937729851, max timestamp=1681937729859Num uniques 1089; 2023-04-19 20:55:29,862 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=339, cellSize=15524, totalHeapSize=35496, min timestamp=1681937729851, max timestamp=1681937729859Num uniques 1089; 2023-04-19 20:55:29,862 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=49874, new segment data size=15524, suffix heap size=141224,new segment heap size=35496 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1089, new segment cells count=339 2023-04-19 20:55:29,862 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=49874, new segment data size=15524, suffix heap size=141224,new segment heap size=35496 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1089, new segment cells count=339 2023-04-19 20:55:29,887 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:29,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 269 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56670 deadline: 1681937789882, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:29,889 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=198, cellSize=9068, totalHeapSize=20832, min timestamp=1681937729868, max timestamp=1681937729886Num uniques 1198; 2023-04-19 20:55:29,889 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=198, cellSize=9058, totalHeapSize=20832, min timestamp=1681937729868, max timestamp=1681937729886Num uniques 1198; 2023-04-19 20:55:29,889 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=54858, new segment data size=9058, suffix heap size=152560,new segment heap size=20832 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1198, new segment cells count=198 2023-04-19 20:55:29,889 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=198, cellSize=9058, totalHeapSize=20832, min timestamp=1681937729868, max timestamp=1681937729886Num uniques 1198; 2023-04-19 20:55:29,889 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=54858, new segment data size=9058, suffix heap size=152560,new segment heap size=20832 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1198, new segment cells count=198 2023-04-19 20:55:29,889 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=54868, new segment data size=9068, suffix heap size=152560,new segment heap size=20832 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1198, new segment cells count=198 2023-04-19 20:55:29,899 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=371 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp/B/9014be03cccd466cb79e8f76ebe0663f 2023-04-19 20:55:29,940 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=307, cellSize=14055, totalHeapSize=32168, min timestamp=1681937729903, max timestamp=1681937729938Num uniques 1057; 2023-04-19 20:55:29,940 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=48405, new segment data size=14055, suffix heap size=137896,new segment heap size=32168 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1057, new segment cells count=307 2023-04-19 20:55:29,940 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=307, cellSize=14055, totalHeapSize=32168, min timestamp=1681937729903, max timestamp=1681937729938Num uniques 1057; 2023-04-19 20:55:29,940 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=48405, new segment data size=14055, suffix heap size=137896,new segment heap size=32168 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1057, new segment cells count=307 2023-04-19 20:55:29,941 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=307, cellSize=14055, totalHeapSize=32168, min timestamp=1681937729903, max timestamp=1681937729938Num uniques 1057; 2023-04-19 20:55:29,941 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=48405, new segment data size=14055, suffix heap size=137896,new segment heap size=32168 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1057, new segment cells count=307 2023-04-19 20:55:29,984 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=166, cellSize=7596, totalHeapSize=17504, min timestamp=1681937729979, max timestamp=1681937729982Num uniques 1166; 2023-04-19 20:55:29,985 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=166, cellSize=7596, totalHeapSize=17504, min timestamp=1681937729979, max timestamp=1681937729982Num uniques 1166; 2023-04-19 20:55:29,985 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=53396, new segment data size=7596, suffix heap size=149232,new segment heap size=17504 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1166, new segment cells count=166 2023-04-19 20:55:29,985 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=53396, new segment data size=7596, suffix heap size=149232,new segment heap size=17504 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1166, new segment cells count=166 2023-04-19 20:55:29,984 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=166, cellSize=7596, totalHeapSize=17504, min timestamp=1681937729979, max timestamp=1681937729982Num uniques 1166; 2023-04-19 20:55:29,985 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=53396, new segment data size=7596, suffix heap size=149232,new segment heap size=17504 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1166, new segment cells count=166 2023-04-19 20:55:30,012 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=275, cellSize=12590, totalHeapSize=28840, min timestamp=1681937730001, max timestamp=1681937730009Num uniques 1025; 2023-04-19 20:55:30,012 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46940, new segment data size=12590, suffix heap size=134568,new segment heap size=28840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1025, new segment cells count=275 2023-04-19 20:55:30,012 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=275, cellSize=12590, totalHeapSize=28840, min timestamp=1681937730001, max timestamp=1681937730009Num uniques 1025; 2023-04-19 20:55:30,012 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=275, cellSize=12590, totalHeapSize=28840, min timestamp=1681937730001, max timestamp=1681937730009Num uniques 1025; 2023-04-19 20:55:30,012 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46940, new segment data size=12590, suffix heap size=134568,new segment heap size=28840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1025, new segment cells count=275 2023-04-19 20:55:30,012 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46940, new segment data size=12590, suffix heap size=134568,new segment heap size=28840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1025, new segment cells count=275 2023-04-19 20:55:30,032 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:30,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 287 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56654 deadline: 1681937790027, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:30,033 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=284, cellSize=13004, totalHeapSize=29776, min timestamp=1681937730020, max timestamp=1681937730031Num uniques 1134; 2023-04-19 20:55:30,033 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=51934, new segment data size=13004, suffix heap size=145904,new segment heap size=29776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1134, new segment cells count=284 2023-04-19 20:55:30,034 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=284, cellSize=13004, totalHeapSize=29776, min timestamp=1681937730020, max timestamp=1681937730031Num uniques 1134; 2023-04-19 20:55:30,034 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=51934, new segment data size=13004, suffix heap size=145904,new segment heap size=29776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1134, new segment cells count=284 2023-04-19 20:55:30,034 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=284, cellSize=13004, totalHeapSize=29776, min timestamp=1681937730020, max timestamp=1681937730031Num uniques 1134; 2023-04-19 20:55:30,034 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=51934, new segment data size=13004, suffix heap size=145904,new segment heap size=29776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1134, new segment cells count=284 2023-04-19 20:55:30,061 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:30,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 320 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56676 deadline: 1681937790055, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:30,062 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=243, cellSize=11128, totalHeapSize=25512, min timestamp=1681937730048, max timestamp=1681937730060Num uniques 1143; 2023-04-19 20:55:30,062 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:30,062 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=52348, new segment data size=11128, suffix heap size=146840,new segment heap size=25512 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1143, new segment cells count=243 2023-04-19 20:55:30,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56610 deadline: 1681937790056, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:30,062 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=243, cellSize=11128, totalHeapSize=25512, min timestamp=1681937730048, max timestamp=1681937730060Num uniques 1143; 2023-04-19 20:55:30,063 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=52348, new segment data size=11128, suffix heap size=146840,new segment heap size=25512 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1143, new segment cells count=243 2023-04-19 20:55:30,063 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=243, cellSize=11128, totalHeapSize=25512, min timestamp=1681937730048, max timestamp=1681937730060Num uniques 1143; 2023-04-19 20:55:30,063 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=52348, new segment data size=11128, suffix heap size=146840,new segment heap size=25512 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1143, new segment cells count=243 2023-04-19 20:55:30,117 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=202, cellSize=9250, totalHeapSize=21248, min timestamp=1681937730091, max timestamp=1681937730115Num uniques 1102; 2023-04-19 20:55:30,117 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=50470, new segment data size=9250, suffix heap size=142576,new segment heap size=21248 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1102, new segment cells count=202 2023-04-19 20:55:30,117 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=202, cellSize=9250, totalHeapSize=21248, min timestamp=1681937730091, max timestamp=1681937730115Num uniques 1102; 2023-04-19 20:55:30,117 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=202, cellSize=9250, totalHeapSize=21248, min timestamp=1681937730091, max timestamp=1681937730115Num uniques 1102; 2023-04-19 20:55:30,118 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=50470, new segment data size=9250, suffix heap size=142576,new segment heap size=21248 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1102, new segment cells count=202 2023-04-19 20:55:30,118 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=50470, new segment data size=9250, suffix heap size=142576,new segment heap size=21248 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1102, new segment cells count=202 2023-04-19 20:55:30,169 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=311, cellSize=14236, totalHeapSize=32584, min timestamp=1681937730137, max timestamp=1681937730168Num uniques 1061; 2023-04-19 20:55:30,170 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=48586, new segment data size=14236, suffix heap size=138312,new segment heap size=32584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1061, new segment cells count=311 2023-04-19 20:55:30,170 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=311, cellSize=14236, totalHeapSize=32584, min timestamp=1681937730137, max timestamp=1681937730168Num uniques 1061; 2023-04-19 20:55:30,170 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=48586, new segment data size=14236, suffix heap size=138312,new segment heap size=32584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1061, new segment cells count=311 2023-04-19 20:55:30,170 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=311, cellSize=14236, totalHeapSize=32584, min timestamp=1681937730137, max timestamp=1681937730168Num uniques 1061; 2023-04-19 20:55:30,171 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=48586, new segment data size=14236, suffix heap size=138312,new segment heap size=32584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1061, new segment cells count=311 2023-04-19 20:55:30,217 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:30,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 295 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56654 deadline: 1681937790207, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:30,219 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=270, cellSize=12360, totalHeapSize=28320, min timestamp=1681937730175, max timestamp=1681937730216Num uniques 1170; 2023-04-19 20:55:30,219 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=53580, new segment data size=12360, suffix heap size=149648,new segment heap size=28320 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1170, new segment cells count=270 2023-04-19 20:55:30,219 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=270, cellSize=12360, totalHeapSize=28320, min timestamp=1681937730175, max timestamp=1681937730216Num uniques 1170; 2023-04-19 20:55:30,219 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=53580, new segment data size=12360, suffix heap size=149648,new segment heap size=28320 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1170, new segment cells count=270 2023-04-19 20:55:30,219 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=270, cellSize=12360, totalHeapSize=28320, min timestamp=1681937730175, max timestamp=1681937730216Num uniques 1170; 2023-04-19 20:55:30,220 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=53580, new segment data size=12360, suffix heap size=149648,new segment heap size=28320 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1170, new segment cells count=270 2023-04-19 20:55:30,250 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:30,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56610 deadline: 1681937790241, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:30,251 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:30,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 330 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56676 deadline: 1681937790247, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:30,251 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=179, cellSize=8184, totalHeapSize=18856, min timestamp=1681937730239, max timestamp=1681937730249Num uniques 1129; 2023-04-19 20:55:30,251 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=51694, new segment data size=8184, suffix heap size=145384,new segment heap size=18856 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1129, new segment cells count=179 2023-04-19 20:55:30,252 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=179, cellSize=8187, totalHeapSize=18856, min timestamp=1681937730239, max timestamp=1681937730249Num uniques 1129; 2023-04-19 20:55:30,252 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=51697, new segment data size=8187, suffix heap size=145384,new segment heap size=18856 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1129, new segment cells count=179 2023-04-19 20:55:30,252 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=179, cellSize=8185, totalHeapSize=18856, min timestamp=1681937730239, max timestamp=1681937730249Num uniques 1129; 2023-04-19 20:55:30,252 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=51695, new segment data size=8185, suffix heap size=145384,new segment heap size=18856 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1129, new segment cells count=179 2023-04-19 20:55:30,307 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=188, cellSize=8608, totalHeapSize=19792, min timestamp=1681937730270, max timestamp=1681937730304Num uniques 1038; 2023-04-19 20:55:30,307 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=188, cellSize=8608, totalHeapSize=19792, min timestamp=1681937730270, max timestamp=1681937730304Num uniques 1038; 2023-04-19 20:55:30,307 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=47538, new segment data size=8608, suffix heap size=135920,new segment heap size=19792 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1038, new segment cells count=188 2023-04-19 20:55:30,307 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=47538, new segment data size=8608, suffix heap size=135920,new segment heap size=19792 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1038, new segment cells count=188 2023-04-19 20:55:30,307 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=188, cellSize=8608, totalHeapSize=19792, min timestamp=1681937730270, max timestamp=1681937730304Num uniques 1038; 2023-04-19 20:55:30,308 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47538, new segment data size=8608, suffix heap size=135920,new segment heap size=19792 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1038, new segment cells count=188 2023-04-19 20:55:30,308 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=371 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp/C/b631298e59a745c79915353c9094e307 2023-04-19 20:55:30,312 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp/A/630b0bf052924e7a912e4e926138960b as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/630b0bf052924e7a912e4e926138960b 2023-04-19 20:55:30,316 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/630b0bf052924e7a912e4e926138960b, entries=50, sequenceid=371, filesize=11.7 K 2023-04-19 20:55:30,316 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp/B/9014be03cccd466cb79e8f76ebe0663f as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/9014be03cccd466cb79e8f76ebe0663f 2023-04-19 20:55:30,320 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/9014be03cccd466cb79e8f76ebe0663f, entries=50, sequenceid=371, filesize=7.1 K 2023-04-19 20:55:30,321 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp/C/b631298e59a745c79915353c9094e307 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/b631298e59a745c79915353c9094e307 2023-04-19 20:55:30,324 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/b631298e59a745c79915353c9094e307, entries=50, sequenceid=371, filesize=7.1 K 2023-04-19 20:55:30,324 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=80.51 KB/82440 for 5067d98ca5f8d3760c5d0eb2b8bb06dc in 1670ms, sequenceid=371, compaction requested=true 2023-04-19 20:55:30,324 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5067d98ca5f8d3760c5d0eb2b8bb06dc: 2023-04-19 20:55:30,324 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:30,325 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:30,325 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:30,325 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:30,325 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:30,325 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:30,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(9158): Flush requested on 5067d98ca5f8d3760c5d0eb2b8bb06dc 2023-04-19 20:55:30,325 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 5067d98ca5f8d3760c5d0eb2b8bb06dc 3/3 column families, dataSize=87.22 KB heapSize=242.81 KB 2023-04-19 20:55:30,326 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5067d98ca5f8d3760c5d0eb2b8bb06dc, store=A 2023-04-19 20:55:30,326 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 20:55:30,326 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5067d98ca5f8d3760c5d0eb2b8bb06dc, store=B 2023-04-19 20:55:30,326 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 20:55:30,326 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5067d98ca5f8d3760c5d0eb2b8bb06dc, store=C 2023-04-19 20:55:30,326 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 20:55:30,326 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:30,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:30,326 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:30,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:30,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:30,326 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:30,327 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:30,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:30,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:30,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:30,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:30,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:30,328 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:30,329 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@16a02806[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:30,329 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:30,329 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:30,330 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:30,330 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:30,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:30,330 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:30,330 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:30,330 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:30,331 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:30,331 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:30,332 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:30,332 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:30,332 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:30,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:30,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:30,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:30,332 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:30,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:30,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:30,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:30,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:30,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:30,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:30,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:30,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:30,335 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:30,335 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:30,335 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:30,335 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:30,335 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:30,335 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:30,338 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:30,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:30,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:30,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:30,338 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041992b89899a1184268bad92eb8fb38621e to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041992b89899a1184268bad92eb8fb38621e 2023-04-19 20:55:30,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:30,339 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:30,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:30,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:30,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:30,340 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:30,340 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:30,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:30,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:30,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:30,342 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:30,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:30,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:30,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:30,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:30,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:30,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:30,344 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:30,344 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:30,344 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:30,344 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:30,344 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:30,344 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:30,345 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:30,345 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:30,345 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:30,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:30,344 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:30,345 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:55:30,345 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:30,345 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:30,345 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:30,345 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:30,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:30,345 DEBUG [Thread-2269] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:30,345 DEBUG [Thread-2269] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:30,359 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:30,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56610 deadline: 1681937790351, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:30,359 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:30,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 332 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56676 deadline: 1681937790351, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:30,365 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:30,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 302 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56654 deadline: 1681937790356, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:30,365 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:30,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56626 deadline: 1681937790357, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:30,368 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:30,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 323 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56670 deadline: 1681937790358, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:30,445 DEBUG [Thread-2269] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:30,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:30,447 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:30,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:30,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:30,448 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:30,448 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:30,448 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:30,448 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:30,449 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:30,449 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:30,450 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:30,450 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:30,450 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:30,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:30,450 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:30,450 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:30,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:30,450 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:30,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:30,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:30,451 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:30,451 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:55:30,451 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:30,451 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:30,451 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:30,451 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:30,451 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:30,451 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:30,451 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:30,451 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. started... 2023-04-19 20:55:30,451 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. as already flushing 2023-04-19 20:55:30,452 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:30,452 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:30,452 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:30,452 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:30,452 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:30,452 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:30,456 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:30,456 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:30,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:30,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:30,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:30,456 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:30,456 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:30,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:30,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:30,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:30,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:30,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:30,457 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:30,457 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:30,457 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@49d4f01a[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:30,457 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:30,459 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:30,459 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:30,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:30,459 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:30,459 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:30,459 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:30,459 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:30,459 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:30,460 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:30,461 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:30,461 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:30,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:30,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:30,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:30,461 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:30,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:30,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:30,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:30,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:30,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:30,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:30,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:30,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:30,463 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:30,463 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:30,463 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:30,463 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:30,463 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:30,463 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:30,464 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:30,464 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:30,464 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:30,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:30,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:30,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:30,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:30,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:30,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:30,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:30,465 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:30,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:30,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:30,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:30,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:30,466 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:30,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:30,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:30,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:30,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:30,474 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:30,475 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:30,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 304 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56654 deadline: 1681937790466, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:30,475 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:30,475 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:30,475 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:30,475 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:30,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:30,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:30,474 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:30,475 DEBUG [Thread-2269] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:30,475 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:55:30,475 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:30,475 DEBUG [Thread-2269] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:30,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56626 deadline: 1681937790466, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:30,475 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:30,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:30,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:30,475 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:30,476 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:30,476 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:30,476 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:30,476 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:30,476 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:30,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 325 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56670 deadline: 1681937790469, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:30,561 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:30,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56610 deadline: 1681937790560, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:30,571 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:30,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 334 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56676 deadline: 1681937790562, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:30,575 DEBUG [Thread-2269] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:30,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:30,577 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:30,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:30,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:30,579 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:30,579 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:30,579 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:30,579 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:30,580 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:30,580 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:30,581 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:30,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:30,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:30,581 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:30,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:30,582 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:30,582 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:30,582 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:30,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:30,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:30,582 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:30,583 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:55:30,583 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:30,583 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:30,583 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:30,583 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:30,583 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:30,583 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:30,583 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:30,583 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. started... 2023-04-19 20:55:30,583 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. as already flushing 2023-04-19 20:55:30,583 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:30,583 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:30,583 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:30,583 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:30,583 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:30,583 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:30,585 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:30,585 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:30,585 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:30,585 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:30,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:30,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:30,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:30,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:30,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:30,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:30,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:30,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:30,587 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:30,587 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e692253[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:30,587 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:30,587 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:30,588 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:30,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:30,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:30,588 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:30,588 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:30,588 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:30,588 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:30,589 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:30,590 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:30,590 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:30,590 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:30,590 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:30,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:30,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:30,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:30,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:30,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:30,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:30,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:30,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:30,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:30,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:30,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:30,598 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:30,598 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:30,598 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:30,598 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:30,598 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:30,598 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:30,600 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:30,600 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:30,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:30,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:30,601 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:30,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:30,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:30,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:30,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:30,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:30,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:30,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:30,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:30,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:30,602 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:30,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:30,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:30,603 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:30,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:30,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:30,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:30,605 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:30,605 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:30,605 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:30,605 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:30,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:30,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:30,605 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:30,605 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:30,606 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:30,605 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:30,606 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:30,606 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:30,606 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:30,606 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:30,606 DEBUG [Thread-2269] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:30,606 DEBUG [Thread-2269] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:30,606 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:30,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:30,678 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:30,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 306 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56654 deadline: 1681937790676, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:30,683 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:30,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 327 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56670 deadline: 1681937790677, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:30,683 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:30,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56626 deadline: 1681937790677, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:30,706 DEBUG [Thread-2269] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:30,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:30,708 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:30,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:30,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:30,709 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:30,709 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:30,709 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:30,709 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:30,710 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:30,710 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:30,712 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:30,712 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:30,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:30,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:30,712 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:30,712 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:30,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:30,713 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:30,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:30,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:30,713 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:30,713 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:55:30,713 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:30,713 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:30,713 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:30,713 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:30,714 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:30,714 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:30,714 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:30,714 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. started... 2023-04-19 20:55:30,714 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. as already flushing 2023-04-19 20:55:30,714 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:30,714 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:30,714 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:30,714 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:30,714 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:30,714 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:30,717 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:30,717 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:30,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:30,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:30,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:30,717 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:30,717 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:30,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:30,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:30,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:30,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:30,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:30,720 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:30,720 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@295da11b[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:30,720 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:30,720 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:30,722 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:30,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:30,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:30,722 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:30,722 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:30,722 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:30,722 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:30,722 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:30,724 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:30,724 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:30,724 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:30,724 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:30,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:30,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:30,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:30,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:30,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:30,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:30,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:30,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:30,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:30,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:30,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:30,727 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:30,727 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:30,727 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:30,727 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:30,727 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:30,727 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:30,729 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:30,729 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:30,729 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:30,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:30,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:30,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:30,729 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:30,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:30,729 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:30,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:30,729 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:30,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:30,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:30,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:30,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:30,730 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:30,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:30,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:30,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:30,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:30,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:30,733 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:30,734 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:30,734 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:30,734 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:30,734 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:30,734 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:30,734 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:30,734 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:30,734 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:30,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:30,734 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:55:30,734 DEBUG [Thread-2269] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:30,734 DEBUG [Thread-2269] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:30,734 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:30,734 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:30,734 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:30,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:30,734 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:30,734 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:30,758 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=936, memsize=29.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp/A/b34d9975f0414ba4936e0ca44220489c 2023-04-19 20:55:30,834 DEBUG [Thread-2269] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:30,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:30,836 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:30,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:30,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:30,838 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:30,838 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:30,838 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:30,838 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:30,838 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:30,838 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:30,840 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:30,840 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:30,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:30,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:30,840 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:30,840 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:30,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:30,841 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:30,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:30,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:30,841 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:30,841 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:55:30,841 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:30,841 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:30,841 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:30,841 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:30,841 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:30,841 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:30,842 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:30,842 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. started... 2023-04-19 20:55:30,842 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. as already flushing 2023-04-19 20:55:30,842 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:30,842 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:30,842 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:30,842 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:30,842 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:30,842 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:30,843 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:30,843 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:30,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:30,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:30,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:30,844 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:30,844 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:30,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:30,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:30,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:30,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:30,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:30,845 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:30,845 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:30,845 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3eaa5c2e[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:30,845 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:30,847 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:30,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:30,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:30,847 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:30,847 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:30,847 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:30,847 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:30,847 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:30,852 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:30,852 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:30,852 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:30,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:30,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:30,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:30,852 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:30,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:30,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:30,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:30,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:30,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:30,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:30,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:30,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:30,854 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:30,854 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:30,854 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:30,854 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:30,854 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:30,854 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:30,856 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:30,856 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:30,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:30,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:30,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:30,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:30,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:30,856 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:30,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:30,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:30,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:30,857 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:30,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:30,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:30,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:30,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:30,858 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:30,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:30,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:30,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:30,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:30,860 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:30,860 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:30,860 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:30,860 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:30,860 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:30,860 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:30,860 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:30,860 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:30,860 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:30,861 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:30,861 DEBUG [Thread-2269] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:30,861 DEBUG [Thread-2269] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:30,861 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:30,861 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:30,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:30,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:30,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:30,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:30,867 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:30,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56610 deadline: 1681937790862, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:30,877 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:30,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 336 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56676 deadline: 1681937790873, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:30,961 DEBUG [Thread-2269] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:30,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:30,963 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:30,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:30,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:30,964 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:30,964 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:30,964 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:30,964 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:30,965 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:30,965 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:30,966 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:30,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:30,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:30,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:30,967 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:30,967 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:30,967 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:30,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:30,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:30,967 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:30,967 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:30,968 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:55:30,968 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:30,968 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:30,968 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:30,968 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:30,968 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:30,968 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:30,968 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:30,968 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. started... 2023-04-19 20:55:30,968 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. as already flushing 2023-04-19 20:55:30,968 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:30,968 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:30,968 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:30,968 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:30,968 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:30,968 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:30,970 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:30,970 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:30,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:30,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:30,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:30,971 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:30,971 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:30,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:30,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:30,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:30,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:30,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:30,972 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:30,972 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@50902fc1[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:30,972 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:30,972 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:30,973 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:30,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:30,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:30,973 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:30,973 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:30,973 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:30,973 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:30,973 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:30,975 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:30,975 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:30,975 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:30,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:30,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:30,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:30,975 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:30,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:30,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:30,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:30,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:30,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:30,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:30,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:30,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:30,977 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:30,977 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:30,977 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:30,977 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:30,977 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:30,977 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:30,979 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:30,979 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:30,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:30,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:30,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:30,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:30,979 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:30,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:30,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:30,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:30,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:30,979 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:30,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:30,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:30,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:30,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:30,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:30,982 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:30,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 308 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56654 deadline: 1681937790980, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:30,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:30,982 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:30,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:30,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:30,985 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:30,985 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:30,986 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:30,986 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:30,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:30,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:30,986 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:30,986 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:30,986 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:30,986 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:30,986 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:30,986 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:30,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:30,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:30,986 DEBUG [Thread-2269] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:30,986 DEBUG [Thread-2269] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:30,986 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:30,986 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:30,989 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:30,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 329 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56670 deadline: 1681937790984, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:30,989 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:30,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56626 deadline: 1681937790985, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:31,086 DEBUG [Thread-2269] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:31,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:31,088 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:31,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:31,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:31,090 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:31,090 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:31,090 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:31,090 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:31,090 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:31,090 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:31,091 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:31,091 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:31,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:31,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:31,092 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:31,092 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:31,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:31,092 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:31,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:31,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:31,093 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:31,093 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:55:31,093 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:31,093 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:31,093 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:31,093 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:31,093 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:31,093 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:31,093 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:31,093 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. started... 2023-04-19 20:55:31,094 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. as already flushing 2023-04-19 20:55:31,094 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:31,094 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:31,094 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:31,094 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:31,094 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:31,094 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:31,095 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:31,095 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:31,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:31,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:31,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:31,096 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:31,096 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:31,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:31,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:31,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:31,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:31,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:31,097 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:31,097 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7bce43a[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:31,097 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:31,097 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:31,098 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:31,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:31,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:31,099 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:31,099 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:31,099 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:31,099 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:31,099 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:31,106 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:31,106 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:31,106 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:31,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:31,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:31,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:31,106 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:31,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:31,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:31,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:31,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:31,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:31,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:31,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:31,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:31,108 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:31,108 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:31,108 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:31,108 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:31,108 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:31,108 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:31,110 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:31,110 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:31,110 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:31,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:31,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:31,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:31,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:31,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:31,111 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:31,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:31,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:31,111 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:31,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:31,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:31,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:31,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:31,112 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:31,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:31,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:31,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:31,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:31,114 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:31,114 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:31,114 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:31,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:31,114 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:31,114 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:55:31,114 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:31,114 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:31,114 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:31,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:31,114 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:31,114 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:31,115 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:31,115 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:31,115 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:31,115 DEBUG [Thread-2269] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:31,115 DEBUG [Thread-2269] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:31,115 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:31,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:31,169 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=936 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp/B/16a796def9844ef086503cfc1ac05681 2023-04-19 20:55:31,215 DEBUG [Thread-2269] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:31,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:31,217 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:31,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:31,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:31,220 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:31,220 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:31,220 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:31,220 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:31,221 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:31,221 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:31,224 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:31,224 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:31,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:31,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:31,224 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:31,224 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:31,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:31,226 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:31,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:31,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:31,227 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:31,240 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:55:31,240 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:31,240 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:31,240 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:31,240 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:31,248 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:31,248 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:31,249 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:31,249 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. started... 2023-04-19 20:55:31,249 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. as already flushing 2023-04-19 20:55:31,249 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:31,249 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:31,249 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:31,249 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:31,249 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:31,249 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:31,252 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:31,252 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:31,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:31,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:31,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:31,253 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:31,253 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:31,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:31,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:31,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:31,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:31,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:31,255 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:31,255 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@432dc8af[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:31,255 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:31,255 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:31,257 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:31,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:31,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:31,257 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:31,257 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:31,257 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:31,258 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:31,258 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:31,259 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:31,259 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:31,259 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:31,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:31,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:31,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:31,259 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:31,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:31,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:31,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:31,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:31,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:31,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:31,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:31,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:31,261 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:31,261 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:31,261 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:31,261 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:31,261 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:31,261 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:31,266 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:31,266 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:31,266 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:31,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:31,267 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:31,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:31,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:31,267 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:31,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:31,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:31,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:31,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:31,269 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:31,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:31,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:31,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:31,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:31,270 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:31,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:31,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:31,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:31,274 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:31,274 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:31,274 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:31,274 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:31,274 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:31,274 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:31,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:31,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:31,274 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:31,274 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:31,274 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:31,274 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:31,274 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:31,274 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:31,274 DEBUG [Thread-2269] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:31,275 DEBUG [Thread-2269] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:31,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:31,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:31,372 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:31,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56610 deadline: 1681937791369, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:31,375 DEBUG [Thread-2269] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:31,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:31,377 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:31,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:31,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:31,378 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:31,378 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:31,378 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:31,378 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:31,379 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:31,379 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:31,380 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:31,380 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:31,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:31,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:31,381 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:31,381 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:31,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:31,382 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:31,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:31,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:31,383 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:31,383 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:55:31,383 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:31,383 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:31,384 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:31,384 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:31,384 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:31,384 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:31,384 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:31,384 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. started... 2023-04-19 20:55:31,384 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. as already flushing 2023-04-19 20:55:31,384 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:31,384 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:31,385 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:31,385 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:31,385 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:31,385 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:31,386 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:31,386 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:31,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:31,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:31,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:31,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:31,387 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:31,387 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:31,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:31,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:31,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:31,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:31,389 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:31,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 338 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56676 deadline: 1681937791382, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:31,392 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:31,392 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2f0ab7c8[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:31,392 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:31,392 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:31,394 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:31,394 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:31,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:31,394 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:31,394 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:31,394 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:31,396 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:31,396 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:31,396 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:31,399 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:31,400 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:31,400 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:31,400 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:31,400 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:31,400 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:31,400 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:31,400 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:31,415 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:31,415 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:31,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:31,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:31,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:31,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:31,416 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:31,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:31,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:31,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:31,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:31,416 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:31,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:31,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:31,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:31,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:31,418 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:31,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:31,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:31,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:31,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:31,424 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:31,424 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:31,424 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:31,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:31,424 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:31,424 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:31,424 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:31,425 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:31,425 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:31,425 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:31,424 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:31,425 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:31,425 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:31,425 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:31,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:31,425 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:31,425 DEBUG [Thread-2269] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:31,425 DEBUG [Thread-2269] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:31,426 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:55:31,492 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:31,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 310 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56654 deadline: 1681937791487, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:31,492 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:31,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56626 deadline: 1681937791490, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:31,496 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:31,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 331 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56670 deadline: 1681937791491, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:31,525 DEBUG [Thread-2269] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:31,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:31,528 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:31,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:31,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:31,529 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:31,529 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:31,529 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:31,529 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:31,529 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:31,529 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:31,531 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:31,531 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:31,531 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:31,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:31,531 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:31,532 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:31,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:31,532 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:31,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:31,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:31,532 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:31,532 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:31,533 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:31,533 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:31,533 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:31,533 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:31,533 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:31,533 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:31,533 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. started... 2023-04-19 20:55:31,533 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. as already flushing 2023-04-19 20:55:31,533 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:31,533 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:31,533 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:31,533 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:31,533 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:31,533 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:31,535 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:31,535 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:31,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:31,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:31,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:31,536 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:31,536 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:31,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:31,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:31,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:31,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:31,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:31,537 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:31,537 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7e356bcf[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:31,537 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:31,538 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:31,539 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:31,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:31,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:31,539 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:31,539 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:31,539 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:31,539 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:31,539 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:31,540 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:31,540 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:31,541 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:31,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:31,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:31,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:31,541 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:31,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:31,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:31,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:31,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:31,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:31,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:31,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:31,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:31,543 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:31,543 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:31,543 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:31,543 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:31,543 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:31,543 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:31,546 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:31,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:31,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:31,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:31,546 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:31,546 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:31,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:31,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:31,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:31,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:31,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:31,547 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:31,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:31,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:31,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:31,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:31,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:31,548 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:31,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:31,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:31,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:31,550 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:31,550 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:31,550 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:31,550 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:31,550 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:31,550 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:31,550 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:31,550 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:31,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:31,550 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:31,550 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:31,550 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:31,551 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:31,551 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:31,551 DEBUG [Thread-2269] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:31,551 DEBUG [Thread-2269] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:31,551 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:31,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:31,580 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=936 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp/C/f3ca9c30a1184edab9fa8fcc03fa5092 2023-04-19 20:55:31,583 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp/A/b34d9975f0414ba4936e0ca44220489c as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/b34d9975f0414ba4936e0ca44220489c 2023-04-19 20:55:31,587 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/b34d9975f0414ba4936e0ca44220489c, entries=250, sequenceid=936, filesize=39.2 K 2023-04-19 20:55:31,588 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp/B/16a796def9844ef086503cfc1ac05681 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/16a796def9844ef086503cfc1ac05681 2023-04-19 20:55:31,592 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/16a796def9844ef086503cfc1ac05681, entries=150, sequenceid=936, filesize=11.9 K 2023-04-19 20:55:31,593 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp/C/f3ca9c30a1184edab9fa8fcc03fa5092 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/f3ca9c30a1184edab9fa8fcc03fa5092 2023-04-19 20:55:31,609 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/f3ca9c30a1184edab9fa8fcc03fa5092, entries=150, sequenceid=936, filesize=11.9 K 2023-04-19 20:55:31,609 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~242.77 KB/248592, currentSize=93.93 KB/96180 for 5067d98ca5f8d3760c5d0eb2b8bb06dc in 1284ms, sequenceid=936, compaction requested=true 2023-04-19 20:55:31,609 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 5067d98ca5f8d3760c5d0eb2b8bb06dc: 2023-04-19 20:55:31,609 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 20:55:31,610 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-19 20:55:31,610 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 20:55:31,610 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-19 20:55:31,611 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-19 20:55:31,612 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 97778 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-19 20:55:31,612 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1912): 5067d98ca5f8d3760c5d0eb2b8bb06dc/A is initiating minor compaction (all files) 2023-04-19 20:55:31,612 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 5067d98ca5f8d3760c5d0eb2b8bb06dc/A in TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:31,612 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/d1a42b823b8f4dd89fb9892c654e087d, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/64e8bbbcd36c465eb635f45f88c903ef, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/630b0bf052924e7a912e4e926138960b, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/b34d9975f0414ba4936e0ca44220489c] into tmpdir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp, totalSize=95.5 K 2023-04-19 20:55:31,612 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting d1a42b823b8f4dd89fb9892c654e087d, keycount=150, bloomtype=ROW, size=26.1 K, encoding=NONE, compression=NONE, seqNum=355, earliestPutTs=1681937727243 2023-04-19 20:55:31,613 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 42062 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-19 20:55:31,613 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1912): 5067d98ca5f8d3760c5d0eb2b8bb06dc/B is initiating minor compaction (all files) 2023-04-19 20:55:31,613 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 64e8bbbcd36c465eb635f45f88c903ef, keycount=100, bloomtype=ROW, size=18.6 K, encoding=NONE, compression=NONE, seqNum=364, earliestPutTs=1681937727900 2023-04-19 20:55:31,613 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5067d98ca5f8d3760c5d0eb2b8bb06dc/B in TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:31,613 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/3df8315ed49142128a18327ccd4909ca, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/76d4439dacdd48548706a4feb1970175, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/9014be03cccd466cb79e8f76ebe0663f, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/16a796def9844ef086503cfc1ac05681] into tmpdir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp, totalSize=41.1 K 2023-04-19 20:55:31,613 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 630b0bf052924e7a912e4e926138960b, keycount=50, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=371, earliestPutTs=1681937728616 2023-04-19 20:55:31,613 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 3df8315ed49142128a18327ccd4909ca, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=355, earliestPutTs=1681937727243 2023-04-19 20:55:31,613 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting b34d9975f0414ba4936e0ca44220489c, keycount=250, bloomtype=ROW, size=39.2 K, encoding=NONE, compression=NONE, seqNum=936, earliestPutTs=1681937730315 2023-04-19 20:55:31,613 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 76d4439dacdd48548706a4feb1970175, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=364, earliestPutTs=1681937727900 2023-04-19 20:55:31,614 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 9014be03cccd466cb79e8f76ebe0663f, keycount=50, bloomtype=ROW, size=7.1 K, encoding=NONE, compression=NONE, seqNum=371, earliestPutTs=1681937728616 2023-04-19 20:55:31,614 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 16a796def9844ef086503cfc1ac05681, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=936, earliestPutTs=1681937730315 2023-04-19 20:55:31,620 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] throttle.PressureAwareThroughputController(145): 5067d98ca5f8d3760c5d0eb2b8bb06dc#B#compaction#366 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 57.69 MB/second 2023-04-19 20:55:31,625 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] throttle.PressureAwareThroughputController(145): 5067d98ca5f8d3760c5d0eb2b8bb06dc#A#compaction#365 average throughput is 2.82 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 57.69 MB/second 2023-04-19 20:55:31,634 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp/B/35bf50208f7e426b8f03120b7923cbb1 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/35bf50208f7e426b8f03120b7923cbb1 2023-04-19 20:55:31,637 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 5067d98ca5f8d3760c5d0eb2b8bb06dc/B of 5067d98ca5f8d3760c5d0eb2b8bb06dc into 35bf50208f7e426b8f03120b7923cbb1(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 20:55:31,637 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5067d98ca5f8d3760c5d0eb2b8bb06dc: 2023-04-19 20:55:31,637 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc., storeName=5067d98ca5f8d3760c5d0eb2b8bb06dc/B, priority=12, startTime=1681937731610; duration=0sec 2023-04-19 20:55:31,637 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 20:55:31,637 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-19 20:55:31,638 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 32354 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-19 20:55:31,638 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1912): 5067d98ca5f8d3760c5d0eb2b8bb06dc/C is initiating minor compaction (all files) 2023-04-19 20:55:31,638 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5067d98ca5f8d3760c5d0eb2b8bb06dc/C in TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:31,638 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/18084385e97d480d9ac6b49db46c2744, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/b631298e59a745c79915353c9094e307, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/f3ca9c30a1184edab9fa8fcc03fa5092] into tmpdir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp, totalSize=31.6 K 2023-04-19 20:55:31,639 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 18084385e97d480d9ac6b49db46c2744, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=364, earliestPutTs=1681937727885 2023-04-19 20:55:31,639 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting b631298e59a745c79915353c9094e307, keycount=50, bloomtype=ROW, size=7.1 K, encoding=NONE, compression=NONE, seqNum=371, earliestPutTs=1681937728616 2023-04-19 20:55:31,639 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting f3ca9c30a1184edab9fa8fcc03fa5092, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=936, earliestPutTs=1681937730315 2023-04-19 20:55:31,644 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] throttle.PressureAwareThroughputController(145): 5067d98ca5f8d3760c5d0eb2b8bb06dc#C#compaction#367 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 57.69 MB/second 2023-04-19 20:55:31,651 DEBUG [Thread-2269] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:31,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:31,653 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:31,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:31,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:31,654 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:31,654 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:31,654 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:31,654 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:31,654 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:31,654 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:31,656 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:31,656 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:31,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:31,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:31,656 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:31,656 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:31,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:31,656 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:31,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:31,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:31,657 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:31,657 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:55:31,657 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:31,657 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:31,657 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:31,657 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:31,660 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:31,660 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:31,660 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:31,660 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. started... 2023-04-19 20:55:31,660 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5067d98ca5f8d3760c5d0eb2b8bb06dc 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-19 20:55:31,660 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5067d98ca5f8d3760c5d0eb2b8bb06dc, store=A 2023-04-19 20:55:31,660 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:55:31,660 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5067d98ca5f8d3760c5d0eb2b8bb06dc, store=B 2023-04-19 20:55:31,660 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:55:31,660 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5067d98ca5f8d3760c5d0eb2b8bb06dc, store=C 2023-04-19 20:55:31,660 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:55:31,677 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041919499162944d4af98a6bd89a717f1bcd to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041919499162944d4af98a6bd89a717f1bcd 2023-04-19 20:55:31,685 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=954, memsize=31.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp/A/45b5898ecfa84bf18388ed2e844643a2 2023-04-19 20:55:32,054 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp/C/2218ea2f6bfc45dd9ef51b2241596fbc as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/2218ea2f6bfc45dd9ef51b2241596fbc 2023-04-19 20:55:32,057 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5067d98ca5f8d3760c5d0eb2b8bb06dc/C of 5067d98ca5f8d3760c5d0eb2b8bb06dc into 2218ea2f6bfc45dd9ef51b2241596fbc(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 20:55:32,058 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5067d98ca5f8d3760c5d0eb2b8bb06dc: 2023-04-19 20:55:32,058 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc., storeName=5067d98ca5f8d3760c5d0eb2b8bb06dc/C, priority=13, startTime=1681937731610; duration=0sec 2023-04-19 20:55:32,058 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 20:55:32,096 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=954 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp/B/3da7f0565e994c93b116ef6abfbfe4ab 2023-04-19 20:55:32,105 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=954 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp/C/67ce81fa975648bf9dcb10a4c009ce00 2023-04-19 20:55:32,109 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp/A/45b5898ecfa84bf18388ed2e844643a2 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/45b5898ecfa84bf18388ed2e844643a2 2023-04-19 20:55:32,112 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/45b5898ecfa84bf18388ed2e844643a2, entries=150, sequenceid=954, filesize=25.4 K 2023-04-19 20:55:32,113 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp/B/3da7f0565e994c93b116ef6abfbfe4ab as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/3da7f0565e994c93b116ef6abfbfe4ab 2023-04-19 20:55:32,116 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/3da7f0565e994c93b116ef6abfbfe4ab, entries=150, sequenceid=954, filesize=11.9 K 2023-04-19 20:55:32,117 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp/C/67ce81fa975648bf9dcb10a4c009ce00 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/67ce81fa975648bf9dcb10a4c009ce00 2023-04-19 20:55:32,120 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/67ce81fa975648bf9dcb10a4c009ce00, entries=150, sequenceid=954, filesize=11.9 K 2023-04-19 20:55:32,121 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=0 B/0 for 5067d98ca5f8d3760c5d0eb2b8bb06dc in 461ms, sequenceid=954, compaction requested=false 2023-04-19 20:55:32,121 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5067d98ca5f8d3760c5d0eb2b8bb06dc: 2023-04-19 20:55:32,121 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:32,121 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:32,121 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:32,121 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:32,121 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:32,121 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:32,123 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:32,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:32,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:32,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:32,123 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:32,123 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:32,123 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:32,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:32,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:32,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:32,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:32,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:32,125 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:32,125 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e48cd07[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:32,125 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:32,125 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:32,127 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:32,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:32,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:32,127 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:32,127 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:32,127 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:32,127 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:32,127 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:32,128 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:32,128 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:32,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:32,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:32,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:32,128 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:32,129 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:32,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:32,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:32,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:32,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:32,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:32,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:32,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:32,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:32,131 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:32,131 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:32,131 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:32,131 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:32,131 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:32,131 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:32,133 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:32,133 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:32,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:32,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:32,133 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:32,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:32,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:32,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:32,134 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:32,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:32,134 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:32,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:32,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:32,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:32,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:32,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:32,135 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:32,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:32,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:32,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:32,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:32,145 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:32,145 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:32,145 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:32,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:32,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:32,145 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:32,145 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:32,145 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:32,145 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:32,145 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:32,145 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:32,145 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:32,145 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:32,145 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:32,145 DEBUG [Thread-2269] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:32,145 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:55:32,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:32,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:32,145 DEBUG [Thread-2269] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:32,246 DEBUG [Thread-2269] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:32,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:32,248 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:32,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:32,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:32,249 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:32,249 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:32,249 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:32,249 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:32,250 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:32,250 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:32,251 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:32,251 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:32,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:32,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:32,251 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:32,251 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:32,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:32,252 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:32,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:32,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:32,252 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:32,252 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:55:32,253 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:32,253 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:32,253 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:32,253 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:32,253 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:32,253 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:32,253 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:32,253 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. started... 2023-04-19 20:55:32,253 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5067d98ca5f8d3760c5d0eb2b8bb06dc: 2023-04-19 20:55:32,253 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:32,253 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:32,253 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:32,253 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:32,253 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:32,253 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:32,255 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:32,255 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:32,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:32,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:32,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:32,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:32,255 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:32,255 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:32,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:32,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:32,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:32,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:32,257 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:32,257 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@338d06d1[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:32,257 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:32,257 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:32,258 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:32,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:32,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:32,258 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:32,258 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:32,258 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:32,258 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:32,258 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:32,261 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:32,261 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:32,261 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:32,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:32,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:32,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:32,261 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:32,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:32,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:32,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:32,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:32,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:32,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:32,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:32,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:32,264 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:32,264 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:32,264 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:32,264 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:32,264 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:32,264 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:32,265 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:32,265 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:32,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:32,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:32,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:32,266 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:32,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:32,266 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:32,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:32,266 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:32,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:32,266 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:32,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:32,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:32,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:32,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:32,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:32,267 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:32,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:32,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:32,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:32,270 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:32,270 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:32,270 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:32,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:32,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:32,270 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:32,270 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:32,270 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:32,270 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:32,270 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:32,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:32,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:32,270 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:32,271 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:32,271 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:32,271 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:32,270 DEBUG [Thread-2269] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:32,271 DEBUG [Thread-2269] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:32,371 DEBUG [Thread-2269] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:32,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:32,374 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:32,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:32,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:32,375 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:32,375 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:32,375 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:32,375 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:32,376 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:32,376 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:32,377 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:32,377 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:32,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:32,377 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:32,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:32,378 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:32,378 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:32,378 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:32,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:32,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:32,379 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:32,379 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:55:32,379 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:32,379 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:32,379 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:32,379 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:32,379 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:32,379 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:32,379 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:32,379 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. started... 2023-04-19 20:55:32,379 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5067d98ca5f8d3760c5d0eb2b8bb06dc: 2023-04-19 20:55:32,379 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:32,379 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:32,380 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:32,380 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:32,380 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:32,380 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:32,382 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:32,382 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:32,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:32,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:32,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:32,382 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:32,382 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:32,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:32,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:32,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:32,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:32,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:32,384 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:32,384 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@49c3439f[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:32,384 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:32,384 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:32,386 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:32,386 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:32,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:32,386 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:32,386 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:32,387 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:32,387 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:32,387 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:32,389 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:32,389 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:32,389 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:32,389 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:32,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:32,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:32,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:32,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:32,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:32,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:32,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:32,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:32,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:32,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:32,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:32,392 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:32,392 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:32,392 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:32,392 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:32,392 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:32,392 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:32,394 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:32,394 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:32,394 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:32,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:32,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:32,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:32,394 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:32,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:32,394 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:32,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:32,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:32,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:32,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:32,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:32,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:32,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:32,396 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:32,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:32,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:32,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:32,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:32,399 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:32,399 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:32,399 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:32,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:32,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:32,399 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:32,399 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:32,399 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:55:32,400 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:32,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:32,399 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:32,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:32,400 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:32,400 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:32,400 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:32,400 DEBUG [Thread-2269] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:32,400 DEBUG [Thread-2269] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:32,400 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:32,400 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:32,445 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp/A/98f7d974b11c40d39d0310fa791d707a as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/98f7d974b11c40d39d0310fa791d707a 2023-04-19 20:55:32,449 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 5067d98ca5f8d3760c5d0eb2b8bb06dc/A of 5067d98ca5f8d3760c5d0eb2b8bb06dc into 98f7d974b11c40d39d0310fa791d707a(size=26.2 K), total size for store is 51.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 20:55:32,449 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5067d98ca5f8d3760c5d0eb2b8bb06dc: 2023-04-19 20:55:32,449 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc., storeName=5067d98ca5f8d3760c5d0eb2b8bb06dc/A, priority=12, startTime=1681937731609; duration=0sec 2023-04-19 20:55:32,449 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 20:55:32,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(9158): Flush requested on 5067d98ca5f8d3760c5d0eb2b8bb06dc 2023-04-19 20:55:32,481 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 5067d98ca5f8d3760c5d0eb2b8bb06dc 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-19 20:55:32,481 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5067d98ca5f8d3760c5d0eb2b8bb06dc, store=A 2023-04-19 20:55:32,482 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:55:32,482 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5067d98ca5f8d3760c5d0eb2b8bb06dc, store=B 2023-04-19 20:55:32,482 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:55:32,482 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5067d98ca5f8d3760c5d0eb2b8bb06dc, store=C 2023-04-19 20:55:32,482 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:55:32,500 DEBUG [Thread-2269] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:32,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:32,504 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:32,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:32,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:32,506 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:32,506 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:32,506 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:32,506 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:32,506 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:32,506 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:32,507 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:32,507 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:32,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:32,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:32,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:32,508 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:32,508 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:32,508 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:32,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:32,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:32,509 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:32,509 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:55:32,509 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:32,509 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:32,509 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:32,509 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:32,510 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:32,510 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:32,510 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:32,511 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. started... 2023-04-19 20:55:32,511 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. as already flushing 2023-04-19 20:55:32,511 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:32,511 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:32,511 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:32,511 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:32,511 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:32,511 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:32,513 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:32,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:32,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:32,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:32,513 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:32,513 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:32,513 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:32,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:32,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:32,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:32,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:32,516 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230419b3fdc6cbed934313b41ec3bbe1fee13f to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419b3fdc6cbed934313b41ec3bbe1fee13f 2023-04-19 20:55:32,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:32,516 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:32,516 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1c7d227b[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:32,516 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:32,517 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:32,518 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:32,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:32,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:32,518 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:32,518 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:32,518 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:32,518 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:32,518 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:32,521 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:32,521 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:32,521 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:32,523 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:32,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:32,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:32,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:32,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:32,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:32,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:32,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:32,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:32,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:32,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:32,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:32,532 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:32,532 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:32,532 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:32,532 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:32,532 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:32,532 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:32,538 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:32,538 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:32,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:32,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:32,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:32,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:32,539 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681937732511, max timestamp=1681937732537Num uniques 859; 2023-04-19 20:55:32,539 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681937732511, max timestamp=1681937732537Num uniques 859; 2023-04-19 20:55:32,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:32,539 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:32,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:32,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:32,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:32,539 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-19 20:55:32,539 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-19 20:55:32,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:32,540 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:32,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:32,541 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681937732511, max timestamp=1681937732537Num uniques 859; 2023-04-19 20:55:32,541 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-19 20:55:32,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:32,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:32,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:32,541 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:32,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:32,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:32,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:32,547 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:32,547 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:32,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:32,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:32,547 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:32,547 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:55:32,547 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:32,547 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:32,547 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:32,547 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:32,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:32,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:32,548 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:32,548 DEBUG [Thread-2269] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:32,548 DEBUG [Thread-2269] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:32,548 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:32,548 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:32,548 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:32,548 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:32,586 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:32,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 351 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56676 deadline: 1681937792569, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:32,586 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:32,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56610 deadline: 1681937792570, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:32,587 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:32,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56626 deadline: 1681937792572, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:32,587 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:32,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 317 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56654 deadline: 1681937792575, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:32,594 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:32,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 338 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56670 deadline: 1681937792586, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:32,648 DEBUG [Thread-2269] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:32,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:32,651 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:32,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:32,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:32,652 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:32,652 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:32,652 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:32,652 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:32,652 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:32,653 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:32,655 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:32,655 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:32,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:32,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:32,655 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:32,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:32,655 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:32,656 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:32,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:32,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:32,656 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:32,657 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:55:32,657 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:32,657 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:32,657 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:32,657 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:32,657 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:32,657 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:32,657 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:32,657 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. started... 2023-04-19 20:55:32,657 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. as already flushing 2023-04-19 20:55:32,657 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:32,657 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:32,657 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:32,657 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:32,657 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:32,657 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:32,659 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:32,659 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:32,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:32,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:32,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:32,659 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:32,659 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:32,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:32,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:32,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:32,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:32,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:32,661 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:32,661 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4485f406[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:32,661 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:32,661 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:32,663 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:32,663 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:32,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:32,663 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:32,663 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:32,663 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:32,663 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:32,663 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:32,664 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:32,664 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:32,664 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:32,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:32,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:32,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:32,664 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:32,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:32,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:32,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:32,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:32,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:32,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:32,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:32,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:32,668 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:32,668 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:32,668 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:32,668 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:32,668 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:32,668 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:32,671 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:32,671 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:32,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:32,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:32,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:32,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:32,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:32,671 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:32,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:32,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:32,671 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:32,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:32,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:32,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:32,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:32,672 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:32,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:32,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:32,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:32,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:32,675 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:32,675 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:32,675 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:32,675 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:32,675 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:32,675 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:32,675 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:32,675 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:32,675 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:32,675 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:32,675 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:55:32,675 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:32,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:32,675 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:32,676 DEBUG [Thread-2269] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:32,676 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:32,676 DEBUG [Thread-2269] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:32,676 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:32,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:32,698 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:32,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 353 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56676 deadline: 1681937792687, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:32,698 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:32,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56610 deadline: 1681937792687, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:32,698 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:32,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56626 deadline: 1681937792688, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:32,698 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:32,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 319 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56654 deadline: 1681937792688, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:32,703 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:32,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 340 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56670 deadline: 1681937792694, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:32,776 DEBUG [Thread-2269] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:32,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:32,778 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:32,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:32,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:32,779 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:32,779 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:32,779 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:32,779 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:32,780 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:32,780 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:32,781 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:32,781 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:32,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:32,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:32,781 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:32,781 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:32,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:32,782 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:32,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:32,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:32,782 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:32,782 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:55:32,782 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:32,782 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:32,782 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:32,783 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:32,783 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:32,783 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:32,783 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:32,783 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. started... 2023-04-19 20:55:32,783 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. as already flushing 2023-04-19 20:55:32,783 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:32,783 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:32,783 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:32,783 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:32,783 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:32,783 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:32,784 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:32,785 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:32,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:32,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:32,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:32,785 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:32,785 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:32,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:32,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:32,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:32,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:32,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:32,786 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:32,786 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1f65d9c9[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:32,786 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:32,786 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:32,788 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:32,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:32,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:32,788 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:32,788 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:32,788 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:32,788 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:32,788 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:32,789 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:32,789 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:32,789 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:32,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:32,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:32,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:32,789 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:32,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:32,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:32,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:32,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:32,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:32,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:32,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:32,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:32,792 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:32,792 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:32,792 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:32,792 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:32,792 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:32,792 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:32,794 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:32,794 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:32,794 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:32,794 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:32,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:32,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:32,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:32,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:32,794 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:32,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:32,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:32,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:32,794 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:32,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:32,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:32,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:32,796 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:32,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:32,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:32,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:32,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:32,798 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:32,798 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:32,798 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:32,798 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:32,798 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:32,798 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:32,798 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:32,798 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:32,798 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:32,798 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:32,798 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:32,798 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:32,798 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:55:32,798 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:32,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:32,798 DEBUG [Thread-2269] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:32,798 DEBUG [Thread-2269] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:32,799 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:32,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:32,899 DEBUG [Thread-2269] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:32,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:32,901 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:32,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:32,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:32,902 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:32,902 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:32,902 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:32,902 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:32,902 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:32,902 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:32,904 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:32,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 355 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56676 deadline: 1681937792899, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:32,905 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:32,905 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:32,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:32,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:32,905 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:32,905 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:32,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:32,906 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:32,906 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:32,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56610 deadline: 1681937792899, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:32,906 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:32,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:32,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:32,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56626 deadline: 1681937792900, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:32,906 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:32,906 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:32,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 321 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56654 deadline: 1681937792900, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:32,907 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:55:32,907 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:32,907 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:32,907 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:32,907 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:32,907 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:32,907 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:32,907 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:32,907 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. started... 2023-04-19 20:55:32,908 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. as already flushing 2023-04-19 20:55:32,908 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:32,908 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:32,908 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:32,908 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:32,908 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:32,908 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:32,908 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:32,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 342 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56670 deadline: 1681937792904, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:32,909 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:32,909 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:32,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:32,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:32,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:32,910 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:32,910 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:32,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:32,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:32,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:32,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:32,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:32,911 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:32,911 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:32,911 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4c69ae01[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:32,911 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:32,912 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:32,912 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:32,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:32,912 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:32,912 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:32,912 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:32,912 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:32,912 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:32,915 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:32,915 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:32,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:32,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:32,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:32,915 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:32,915 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:32,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:32,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:32,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:32,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:32,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:32,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:32,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:32,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:32,917 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:32,917 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:32,917 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:32,917 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:32,917 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:32,917 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:32,920 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:32,920 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:32,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:32,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:32,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:32,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:32,920 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:32,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:32,920 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:32,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:32,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:32,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:32,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:32,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:32,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:32,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:32,921 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:32,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:32,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:32,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:32,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:32,923 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:32,923 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:32,923 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:32,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:32,924 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:32,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:32,924 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:32,924 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:32,924 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:32,924 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:32,924 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:32,924 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:32,924 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:32,924 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:32,924 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:55:32,925 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:32,925 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:32,925 DEBUG [Thread-2269] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:32,925 DEBUG [Thread-2269] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:32,954 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=968, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp/A/6ac196ad591443668c5308f0fd40bf62 2023-04-19 20:55:33,025 DEBUG [Thread-2269] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:33,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:33,028 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:33,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:33,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:33,029 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:33,029 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:33,029 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:33,029 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:33,030 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:33,030 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:33,031 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:33,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:33,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:33,031 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:33,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:33,032 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:33,032 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:33,032 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:33,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:33,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:33,032 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:33,032 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:55:33,032 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:33,032 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:33,032 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:33,033 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:33,033 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:33,033 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:33,033 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:33,033 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. started... 2023-04-19 20:55:33,033 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. as already flushing 2023-04-19 20:55:33,033 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:33,033 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:33,033 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:33,033 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:33,033 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:33,033 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:33,043 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:33,043 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:33,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:33,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:33,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:33,043 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:33,043 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:33,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:33,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:33,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:33,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:33,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:33,044 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:33,044 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@39f2330d[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:33,044 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:33,045 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:33,046 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:33,046 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:33,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:33,046 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:33,046 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:33,046 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:33,046 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:33,046 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:33,047 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:33,047 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:33,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:33,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:33,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:33,047 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:33,047 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:33,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:33,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:33,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:33,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:33,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:33,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:33,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:33,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:33,049 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:33,049 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:33,049 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:33,049 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:33,049 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:33,050 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:33,052 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:33,052 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:33,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:33,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:33,052 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:33,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:33,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:33,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:33,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:33,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:33,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:33,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:33,053 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:33,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:33,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:33,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:33,054 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:33,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:33,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:33,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:33,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:33,056 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:33,056 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:33,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:33,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:33,056 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:33,056 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:33,056 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:33,057 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:33,057 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:33,057 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:33,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:33,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:33,057 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:33,057 DEBUG [Thread-2269] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:33,057 DEBUG [Thread-2269] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:33,057 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:33,057 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:33,057 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:33,157 DEBUG [Thread-2269] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:33,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:33,159 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:33,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:33,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:33,161 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:33,161 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:33,161 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:33,161 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:33,161 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:33,161 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:33,163 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:33,163 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:33,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:33,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:33,163 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:33,163 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:33,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:33,163 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:33,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:33,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:33,164 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:33,164 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:55:33,164 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:33,164 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:33,164 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:33,164 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:33,164 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:33,165 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:33,165 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:33,165 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. started... 2023-04-19 20:55:33,165 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. as already flushing 2023-04-19 20:55:33,165 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:33,165 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:33,165 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:33,165 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:33,165 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:33,165 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:33,166 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:33,166 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:33,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:33,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:33,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:33,167 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:33,167 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:33,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:33,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:33,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:33,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:33,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:33,168 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:33,169 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@523cc5f1[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:33,169 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:33,169 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:33,172 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:33,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:33,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:33,172 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:33,172 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:33,172 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:33,172 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:33,172 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:33,173 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:33,173 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:33,174 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:33,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:33,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:33,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:33,174 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:33,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:33,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:33,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:33,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:33,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:33,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:33,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:33,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:33,176 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:33,176 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:33,176 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:33,176 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:33,176 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:33,176 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:33,178 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:33,178 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:33,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:33,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:33,178 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:33,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:33,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:33,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:33,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:33,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:33,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:33,178 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:33,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:33,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:33,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:33,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:33,180 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:33,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:33,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:33,183 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:33,183 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:33,183 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:33,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:33,183 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:33,183 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:33,183 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:33,183 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:33,183 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:33,183 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:33,183 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:33,184 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:33,184 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:33,184 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:33,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:33,184 DEBUG [Thread-2269] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:33,184 DEBUG [Thread-2269] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:33,184 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:33,184 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:55:33,211 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:33,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 357 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56676 deadline: 1681937793206, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:33,213 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:33,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56626 deadline: 1681937793207, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:33,214 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:33,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56610 deadline: 1681937793208, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:33,214 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:33,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 323 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56654 deadline: 1681937793208, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:33,215 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:33,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 344 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56670 deadline: 1681937793211, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:33,284 DEBUG [Thread-2269] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:33,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:33,286 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:33,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:33,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:33,287 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:33,287 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:33,288 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:33,288 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:33,288 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:33,288 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:33,289 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:33,289 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:33,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:33,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:33,290 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:33,290 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:33,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:33,290 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:33,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:33,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:33,291 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:33,291 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:55:33,291 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:33,291 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:33,291 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:33,291 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:33,291 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:33,291 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:33,291 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:33,291 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. started... 2023-04-19 20:55:33,292 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. as already flushing 2023-04-19 20:55:33,292 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:33,292 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:33,292 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:33,292 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:33,292 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:33,292 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:33,293 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:33,293 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:33,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:33,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:33,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:33,294 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:33,294 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:33,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:33,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:33,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:33,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:33,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:33,295 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:33,295 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:33,295 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6326f5ee[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:33,295 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:33,298 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:33,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:33,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:33,298 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:33,298 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:33,298 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:33,298 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:33,298 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:33,300 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:33,300 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:33,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:33,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:33,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:33,300 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:33,300 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:33,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:33,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:33,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:33,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:33,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:33,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:33,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:33,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:33,302 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:33,302 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:33,302 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:33,302 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:33,302 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:33,302 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:33,304 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:33,304 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:33,304 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:33,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:33,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:33,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:33,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:33,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:33,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:33,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:33,305 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:33,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:33,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:33,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:33,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:33,306 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:33,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:33,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:33,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:33,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:33,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:33,308 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:33,308 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:33,308 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:33,308 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:33,308 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:33,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:33,308 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:33,308 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:33,308 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:33,308 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:33,309 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:33,309 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:33,309 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:33,309 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:33,309 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:55:33,309 DEBUG [Thread-2269] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:33,309 DEBUG [Thread-2269] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:33,309 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:33,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:33,369 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=968 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp/B/e5bdf32d0342460eb4fb5bc4e2569b37 2023-04-19 20:55:33,409 DEBUG [Thread-2269] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:33,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:33,412 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:33,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:33,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:33,413 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:33,413 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:33,413 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:33,413 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:33,414 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:33,414 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:33,416 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:33,416 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:33,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:33,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:33,416 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:33,416 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:33,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:33,416 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:33,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:33,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:33,417 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:33,424 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:55:33,424 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:33,424 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:33,424 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:33,424 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:33,424 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:33,424 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:33,424 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:33,424 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. started... 2023-04-19 20:55:33,424 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. as already flushing 2023-04-19 20:55:33,425 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:33,425 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:33,425 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:33,425 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:33,425 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:33,425 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:33,426 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:33,426 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:33,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:33,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:33,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:33,426 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:33,426 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:33,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:33,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:33,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:33,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:33,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:33,428 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:33,428 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:33,428 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@623e562d[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:33,428 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:33,430 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:33,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:33,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:33,430 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:33,430 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:33,430 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:33,430 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:33,430 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:33,434 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:33,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:33,435 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:33,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:33,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:33,435 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:33,435 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:33,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:33,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:33,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:33,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:33,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:33,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:33,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:33,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:33,437 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:33,437 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:33,437 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:33,437 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:33,437 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:33,437 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:33,439 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:33,439 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:33,439 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:33,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:33,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:33,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:33,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:33,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:33,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:33,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:33,440 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:33,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:33,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:33,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:33,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:33,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:33,441 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:33,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:33,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:33,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:33,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:33,443 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:33,443 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:33,443 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:33,443 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:33,443 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:33,443 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:33,443 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:33,443 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:33,443 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:33,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:33,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:33,444 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:33,444 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:33,444 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:33,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:33,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:33,445 DEBUG [Thread-2269] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:33,445 DEBUG [Thread-2269] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:33,545 DEBUG [Thread-2269] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:33,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:33,548 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:33,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:33,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:33,549 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:33,549 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:33,549 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:33,549 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:33,549 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:33,549 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:33,557 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:33,557 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:33,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:33,557 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:33,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:33,557 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:33,557 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:33,557 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:33,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:33,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:33,558 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:33,558 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:55:33,558 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:33,558 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:33,558 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:33,558 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:33,559 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:33,559 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:33,559 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:33,559 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. started... 2023-04-19 20:55:33,559 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. as already flushing 2023-04-19 20:55:33,559 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:33,559 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:33,559 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:33,559 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:33,559 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:33,559 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:33,561 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:33,561 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:33,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:33,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:33,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:33,561 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:33,561 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:33,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:33,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:33,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:33,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:33,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:33,563 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:33,563 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4a7c7b3c[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:33,563 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:33,563 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:33,565 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:33,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:33,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:33,565 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:33,565 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:33,565 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:33,567 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:33,567 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:33,568 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:33,568 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:33,569 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:33,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:33,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:33,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:33,569 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:33,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:33,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:33,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:33,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:33,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:33,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:33,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:33,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:33,572 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:33,572 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:33,572 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:33,572 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:33,572 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:33,572 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:33,575 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:33,575 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:33,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:33,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:33,575 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:33,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:33,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:33,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:33,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:33,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:33,576 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:33,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:33,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:33,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:33,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:33,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:33,577 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:33,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:33,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:33,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:33,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:33,580 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:33,580 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:33,580 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:33,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:33,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:33,580 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:33,580 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:33,580 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:33,580 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:33,580 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:33,580 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:33,580 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:33,580 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:33,580 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:33,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:33,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:33,580 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:55:33,580 DEBUG [Thread-2269] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:33,581 DEBUG [Thread-2269] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:33,581 DEBUG [Thread-2276] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x660630e8 to 127.0.0.1:58736 2023-04-19 20:55:33,581 DEBUG [Thread-2276] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-19 20:55:33,582 DEBUG [Thread-2270] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6846dddb to 127.0.0.1:58736 2023-04-19 20:55:33,582 DEBUG [Thread-2270] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-19 20:55:33,584 DEBUG [Thread-2272] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7ea43962 to 127.0.0.1:58736 2023-04-19 20:55:33,584 DEBUG [Thread-2272] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-19 20:55:33,589 DEBUG [Thread-2278] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x793239ee to 127.0.0.1:58736 2023-04-19 20:55:33,590 DEBUG [Thread-2278] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-19 20:55:33,591 DEBUG [Thread-2274] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2abd7961 to 127.0.0.1:58736 2023-04-19 20:55:33,591 DEBUG [Thread-2274] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-19 20:55:33,681 DEBUG [Thread-2269] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:33,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:33,714 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:33,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 359 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56676 deadline: 1681937793714, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:33,715 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:33,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56610 deadline: 1681937793715, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:33,716 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:33,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56626 deadline: 1681937793716, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:33,718 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:33,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 325 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56654 deadline: 1681937793718, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:33,720 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:33,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 346 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56670 deadline: 1681937793720, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:33,787 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=968 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp/C/21265545ee324f04a4496c82b88454bd 2023-04-19 20:55:33,791 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp/A/6ac196ad591443668c5308f0fd40bf62 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/6ac196ad591443668c5308f0fd40bf62 2023-04-19 20:55:33,795 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/6ac196ad591443668c5308f0fd40bf62, entries=250, sequenceid=968, filesize=39.2 K 2023-04-19 20:55:33,795 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp/B/e5bdf32d0342460eb4fb5bc4e2569b37 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/e5bdf32d0342460eb4fb5bc4e2569b37 2023-04-19 20:55:33,798 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/e5bdf32d0342460eb4fb5bc4e2569b37, entries=150, sequenceid=968, filesize=11.9 K 2023-04-19 20:55:33,799 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp/C/21265545ee324f04a4496c82b88454bd as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/21265545ee324f04a4496c82b88454bd 2023-04-19 20:55:33,802 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/21265545ee324f04a4496c82b88454bd, entries=150, sequenceid=968, filesize=11.9 K 2023-04-19 20:55:33,802 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 5067d98ca5f8d3760c5d0eb2b8bb06dc in 1321ms, sequenceid=968, compaction requested=true 2023-04-19 20:55:33,803 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 5067d98ca5f8d3760c5d0eb2b8bb06dc: 2023-04-19 20:55:33,803 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 20:55:33,803 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 20:55:33,803 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-19 20:55:33,803 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-19 20:55:33,803 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 20:55:33,804 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 93006 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-19 20:55:33,804 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1912): 5067d98ca5f8d3760c5d0eb2b8bb06dc/A is initiating minor compaction (all files) 2023-04-19 20:55:33,804 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37342 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-19 20:55:33,804 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 5067d98ca5f8d3760c5d0eb2b8bb06dc/A in TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:33,804 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1912): 5067d98ca5f8d3760c5d0eb2b8bb06dc/B is initiating minor compaction (all files) 2023-04-19 20:55:33,804 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/98f7d974b11c40d39d0310fa791d707a, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/45b5898ecfa84bf18388ed2e844643a2, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/6ac196ad591443668c5308f0fd40bf62] into tmpdir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp, totalSize=90.8 K 2023-04-19 20:55:33,804 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5067d98ca5f8d3760c5d0eb2b8bb06dc/B in TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:33,804 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/35bf50208f7e426b8f03120b7923cbb1, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/3da7f0565e994c93b116ef6abfbfe4ab, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/e5bdf32d0342460eb4fb5bc4e2569b37] into tmpdir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp, totalSize=36.5 K 2023-04-19 20:55:33,804 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 98f7d974b11c40d39d0310fa791d707a, keycount=150, bloomtype=ROW, size=26.2 K, encoding=NONE, compression=NONE, seqNum=936, earliestPutTs=1681937730315 2023-04-19 20:55:33,804 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 35bf50208f7e426b8f03120b7923cbb1, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=936, earliestPutTs=1681937730315 2023-04-19 20:55:33,804 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 45b5898ecfa84bf18388ed2e844643a2, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=954, earliestPutTs=1681937730347 2023-04-19 20:55:33,804 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 3da7f0565e994c93b116ef6abfbfe4ab, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=954, earliestPutTs=1681937730347 2023-04-19 20:55:33,805 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 6ac196ad591443668c5308f0fd40bf62, keycount=250, bloomtype=ROW, size=39.2 K, encoding=NONE, compression=NONE, seqNum=968, earliestPutTs=1681937732419 2023-04-19 20:55:33,805 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting e5bdf32d0342460eb4fb5bc4e2569b37, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=968, earliestPutTs=1681937732419 2023-04-19 20:55:33,819 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] throttle.PressureAwareThroughputController(145): 5067d98ca5f8d3760c5d0eb2b8bb06dc#B#compaction#374 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 57.69 MB/second 2023-04-19 20:55:33,826 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] throttle.PressureAwareThroughputController(145): 5067d98ca5f8d3760c5d0eb2b8bb06dc#A#compaction#375 average throughput is 2.82 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 57.69 MB/second 2023-04-19 20:55:34,234 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp/B/a7f6696a3f77400284473332299b9843 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/a7f6696a3f77400284473332299b9843 2023-04-19 20:55:34,238 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5067d98ca5f8d3760c5d0eb2b8bb06dc/B of 5067d98ca5f8d3760c5d0eb2b8bb06dc into a7f6696a3f77400284473332299b9843(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 20:55:34,238 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5067d98ca5f8d3760c5d0eb2b8bb06dc: 2023-04-19 20:55:34,238 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc., storeName=5067d98ca5f8d3760c5d0eb2b8bb06dc/B, priority=13, startTime=1681937733803; duration=0sec 2023-04-19 20:55:34,238 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 20:55:34,239 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-19 20:55:34,239 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37342 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-19 20:55:34,240 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1912): 5067d98ca5f8d3760c5d0eb2b8bb06dc/C is initiating minor compaction (all files) 2023-04-19 20:55:34,240 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5067d98ca5f8d3760c5d0eb2b8bb06dc/C in TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:34,240 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/2218ea2f6bfc45dd9ef51b2241596fbc, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/67ce81fa975648bf9dcb10a4c009ce00, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/21265545ee324f04a4496c82b88454bd] into tmpdir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp, totalSize=36.5 K 2023-04-19 20:55:34,240 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 2218ea2f6bfc45dd9ef51b2241596fbc, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=936, earliestPutTs=1681937730315 2023-04-19 20:55:34,242 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 67ce81fa975648bf9dcb10a4c009ce00, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=954, earliestPutTs=1681937730347 2023-04-19 20:55:34,243 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 21265545ee324f04a4496c82b88454bd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=968, earliestPutTs=1681937732419 2023-04-19 20:55:34,249 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] throttle.PressureAwareThroughputController(145): 5067d98ca5f8d3760c5d0eb2b8bb06dc#C#compaction#376 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 57.69 MB/second 2023-04-19 20:55:34,266 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp/C/d42cfa1dc0614d05a925426ce09cc4e7 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/d42cfa1dc0614d05a925426ce09cc4e7 2023-04-19 20:55:34,270 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5067d98ca5f8d3760c5d0eb2b8bb06dc/C of 5067d98ca5f8d3760c5d0eb2b8bb06dc into d42cfa1dc0614d05a925426ce09cc4e7(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 20:55:34,270 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5067d98ca5f8d3760c5d0eb2b8bb06dc: 2023-04-19 20:55:34,270 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc., storeName=5067d98ca5f8d3760c5d0eb2b8bb06dc/C, priority=13, startTime=1681937733803; duration=0sec 2023-04-19 20:55:34,270 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 20:55:34,668 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp/A/c6730b93fc0442728faf7d36918c6231 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/c6730b93fc0442728faf7d36918c6231 2023-04-19 20:55:34,672 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5067d98ca5f8d3760c5d0eb2b8bb06dc/A of 5067d98ca5f8d3760c5d0eb2b8bb06dc into c6730b93fc0442728faf7d36918c6231(size=26.3 K), total size for store is 26.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 20:55:34,672 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5067d98ca5f8d3760c5d0eb2b8bb06dc: 2023-04-19 20:55:34,672 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc., storeName=5067d98ca5f8d3760c5d0eb2b8bb06dc/A, priority=13, startTime=1681937733803; duration=0sec 2023-04-19 20:55:34,672 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 20:55:34,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(9158): Flush requested on 5067d98ca5f8d3760c5d0eb2b8bb06dc 2023-04-19 20:55:34,718 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 5067d98ca5f8d3760c5d0eb2b8bb06dc 3/3 column families, dataSize=140.89 KB heapSize=395.53 KB 2023-04-19 20:55:34,718 DEBUG [Thread-2263] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5b7d4a51 to 127.0.0.1:58736 2023-04-19 20:55:34,718 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5067d98ca5f8d3760c5d0eb2b8bb06dc, store=A 2023-04-19 20:55:34,718 DEBUG [Thread-2263] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-19 20:55:34,718 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 20:55:34,718 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5067d98ca5f8d3760c5d0eb2b8bb06dc, store=B 2023-04-19 20:55:34,718 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 20:55:34,718 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5067d98ca5f8d3760c5d0eb2b8bb06dc, store=C 2023-04-19 20:55:34,718 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 20:55:34,721 DEBUG [Thread-2259] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2deaf1fa to 127.0.0.1:58736 2023-04-19 20:55:34,721 DEBUG [Thread-2259] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-19 20:55:34,779 DEBUG [Thread-2265] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4335c1f2 to 127.0.0.1:58736 2023-04-19 20:55:34,779 DEBUG [Thread-2265] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-19 20:55:34,779 DEBUG [Thread-2261] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6e7e18ef to 127.0.0.1:58736 2023-04-19 20:55:34,780 DEBUG [Thread-2261] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-19 20:55:34,780 DEBUG [Thread-2267] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6bbc9fc8 to 127.0.0.1:58736 2023-04-19 20:55:34,780 DEBUG [Thread-2267] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-19 20:55:34,780 INFO [Listener at localhost/34893] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-19 20:55:34,780 INFO [Listener at localhost/34893] hbase.AcidGuaranteesTestTool(394): wrote 206 2023-04-19 20:55:34,780 INFO [Listener at localhost/34893] hbase.AcidGuaranteesTestTool(394): wrote 231 2023-04-19 20:55:34,780 INFO [Listener at localhost/34893] hbase.AcidGuaranteesTestTool(394): wrote 246 2023-04-19 20:55:34,780 INFO [Listener at localhost/34893] hbase.AcidGuaranteesTestTool(394): wrote 139 2023-04-19 20:55:34,780 INFO [Listener at localhost/34893] hbase.AcidGuaranteesTestTool(394): wrote 96 2023-04-19 20:55:34,780 INFO [Listener at localhost/34893] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-19 20:55:34,781 INFO [Listener at localhost/34893] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-19 20:55:34,781 INFO [Listener at localhost/34893] hbase.AcidGuaranteesTestTool(402): scanned 1684 2023-04-19 20:55:34,781 INFO [Listener at localhost/34893] hbase.AcidGuaranteesTestTool(403): verified 5051 rows 2023-04-19 20:55:34,781 INFO [Listener at localhost/34893] hbase.AcidGuaranteesTestTool(402): scanned 1686 2023-04-19 20:55:34,781 INFO [Listener at localhost/34893] hbase.AcidGuaranteesTestTool(403): verified 5048 rows 2023-04-19 20:55:34,781 INFO [Listener at localhost/34893] hbase.AcidGuaranteesTestTool(402): scanned 1680 2023-04-19 20:55:34,781 INFO [Listener at localhost/34893] hbase.AcidGuaranteesTestTool(403): verified 5038 rows 2023-04-19 20:55:34,781 INFO [Listener at localhost/34893] hbase.AcidGuaranteesTestTool(402): scanned 1686 2023-04-19 20:55:34,781 INFO [Listener at localhost/34893] hbase.AcidGuaranteesTestTool(403): verified 5048 rows 2023-04-19 20:55:34,781 INFO [Listener at localhost/34893] hbase.AcidGuaranteesTestTool(402): scanned 1701 2023-04-19 20:55:34,781 INFO [Listener at localhost/34893] hbase.AcidGuaranteesTestTool(403): verified 5102 rows 2023-04-19 20:55:34,781 INFO [Listener at localhost/34893] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-19 20:55:34,781 DEBUG [Listener at localhost/34893] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x118e131f to 127.0.0.1:58736 2023-04-19 20:55:34,781 DEBUG [Listener at localhost/34893] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-19 20:55:34,783 INFO [Listener at localhost/34893] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-19 20:55:34,783 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-19 20:55:34,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure2.ProcedureExecutor(1029): Stored pid=43, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-19 20:55:34,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-19 20:55:34,788 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681937734788"}]},"ts":"1681937734788"} 2023-04-19 20:55:34,790 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-19 20:55:34,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-19 20:55:34,889 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-19 20:55:34,890 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=44, ppid=43, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=5067d98ca5f8d3760c5d0eb2b8bb06dc, UNASSIGN}] 2023-04-19 20:55:34,892 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, ppid=43, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=5067d98ca5f8d3760c5d0eb2b8bb06dc, UNASSIGN 2023-04-19 20:55:34,892 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=44 updating hbase:meta row=5067d98ca5f8d3760c5d0eb2b8bb06dc, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:34,892 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681937734892"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681937734892"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681937734892"}]},"ts":"1681937734892"} 2023-04-19 20:55:34,893 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=45, ppid=44, state=RUNNABLE; CloseRegionProcedure 5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971}] 2023-04-19 20:55:35,045 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 5067d98ca5f8d3760c5d0eb2b8bb06dc 2023-04-19 20:55:35,045 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 5067d98ca5f8d3760c5d0eb2b8bb06dc, disabling compactions & flushes 2023-04-19 20:55:35,045 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1858): waiting for 0 compactions & cache flush to complete for region TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:35,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-19 20:55:35,138 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230419781ad46d1e924141b8b426b70b51d8ca to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419781ad46d1e924141b8b426b70b51d8ca 2023-04-19 20:55:35,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-19 20:55:35,545 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1007, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp/A/df7681c8951b43f5882299ca662ceb3a 2023-04-19 20:55:35,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-19 20:55:35,953 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=1007 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp/B/b8bb0dcf88e1466a92d87da018aab604 2023-04-19 20:55:36,363 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=1007 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp/C/44a93841c8484e7a91f293cf83b2d0c1 2023-04-19 20:55:36,367 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp/A/df7681c8951b43f5882299ca662ceb3a as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/df7681c8951b43f5882299ca662ceb3a 2023-04-19 20:55:36,370 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/df7681c8951b43f5882299ca662ceb3a, entries=150, sequenceid=1007, filesize=25.4 K 2023-04-19 20:55:36,370 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp/B/b8bb0dcf88e1466a92d87da018aab604 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/b8bb0dcf88e1466a92d87da018aab604 2023-04-19 20:55:36,374 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/b8bb0dcf88e1466a92d87da018aab604, entries=150, sequenceid=1007, filesize=11.9 K 2023-04-19 20:55:36,374 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp/C/44a93841c8484e7a91f293cf83b2d0c1 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/44a93841c8484e7a91f293cf83b2d0c1 2023-04-19 20:55:36,378 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/44a93841c8484e7a91f293cf83b2d0c1, entries=150, sequenceid=1007, filesize=11.9 K 2023-04-19 20:55:36,378 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~395.48 KB/404976, currentSize=26.84 KB/27480 for 5067d98ca5f8d3760c5d0eb2b8bb06dc in 1660ms, sequenceid=1007, compaction requested=false 2023-04-19 20:55:36,378 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 5067d98ca5f8d3760c5d0eb2b8bb06dc: 2023-04-19 20:55:36,378 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:36,378 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:36,378 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. after waiting 0 ms 2023-04-19 20:55:36,378 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:36,378 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 5067d98ca5f8d3760c5d0eb2b8bb06dc 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-19 20:55:36,378 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5067d98ca5f8d3760c5d0eb2b8bb06dc, store=A 2023-04-19 20:55:36,379 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:55:36,379 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5067d98ca5f8d3760c5d0eb2b8bb06dc, store=B 2023-04-19 20:55:36,379 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:55:36,379 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5067d98ca5f8d3760c5d0eb2b8bb06dc, store=C 2023-04-19 20:55:36,379 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:55:36,790 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304190452a716afcf45499fa9aa9bc0846380 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304190452a716afcf45499fa9aa9bc0846380 2023-04-19 20:55:36,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-19 20:55:37,195 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1014, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp/A/dd27c1508d1e4210b9b89b0ddac58857 2023-04-19 20:55:37,605 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=1014 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp/B/cd30f732e95843f7ada8bdd95136580f 2023-04-19 20:55:38,016 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=1014 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp/C/e7b3f3d4403148cc974610ff1780896c 2023-04-19 20:55:38,019 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp/A/dd27c1508d1e4210b9b89b0ddac58857 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/dd27c1508d1e4210b9b89b0ddac58857 2023-04-19 20:55:38,023 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/dd27c1508d1e4210b9b89b0ddac58857, entries=150, sequenceid=1014, filesize=25.4 K 2023-04-19 20:55:38,023 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp/B/cd30f732e95843f7ada8bdd95136580f as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/cd30f732e95843f7ada8bdd95136580f 2023-04-19 20:55:38,026 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/cd30f732e95843f7ada8bdd95136580f, entries=150, sequenceid=1014, filesize=11.9 K 2023-04-19 20:55:38,026 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/.tmp/C/e7b3f3d4403148cc974610ff1780896c as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/e7b3f3d4403148cc974610ff1780896c 2023-04-19 20:55:38,029 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/e7b3f3d4403148cc974610ff1780896c, entries=150, sequenceid=1014, filesize=11.9 K 2023-04-19 20:55:38,030 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for 5067d98ca5f8d3760c5d0eb2b8bb06dc in 1652ms, sequenceid=1014, compaction requested=true 2023-04-19 20:55:38,053 DEBUG [StoreCloser-TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/7a9c45df3a7241a68697ab5f440c3203, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/ea3c1d41454940329667338ab43ec20c, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/ad1315a26f8e4150b205bc0be6b2463a, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/4d46d96b97ff4ec8b010199c78ab2aeb, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/c062767b9d634d2fbb9737f52c20e15a, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/c8a8147791b546ac8b0cf5eb79a7cc66, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/79df7df7265d446dbcc5096b9e27a89e, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/f354c78a24c445fd9b41bc8987f232d8, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/8133d03912154df89d6084ed7259cdcc, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/98ee35ec3c42466faf36f0d83c221698, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/a373fc3f0b68496894e5b98c7b87c4a2, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/3e5713ebc792425caf9af7d5349a2857, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/ce774494e7894fbab682540dbcbb5957, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/c2e0e314b81742009f61561aade9f5c4, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/c0d0d3c6ca2a463e9091e8ea9a542b4f, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/39089e48a7004a539f5f6177094991ef, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/5586d5e2347a45ee9e8ca97d97422563, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/667c47b0a50540b7afef7896f3bc1e42, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/0c5836c6cce74b26a94c5ea844769ba5, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/d1a42b823b8f4dd89fb9892c654e087d, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/64e8bbbcd36c465eb635f45f88c903ef, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/630b0bf052924e7a912e4e926138960b, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/b34d9975f0414ba4936e0ca44220489c, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/98f7d974b11c40d39d0310fa791d707a, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/45b5898ecfa84bf18388ed2e844643a2, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/6ac196ad591443668c5308f0fd40bf62] to archive 2023-04-19 20:55:38,054 DEBUG [StoreCloser-TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-19 20:55:38,055 DEBUG [StoreCloser-TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/7a9c45df3a7241a68697ab5f440c3203 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/7a9c45df3a7241a68697ab5f440c3203 2023-04-19 20:55:38,056 DEBUG [StoreCloser-TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/ea3c1d41454940329667338ab43ec20c to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/ea3c1d41454940329667338ab43ec20c 2023-04-19 20:55:38,057 DEBUG [StoreCloser-TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/ad1315a26f8e4150b205bc0be6b2463a to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/ad1315a26f8e4150b205bc0be6b2463a 2023-04-19 20:55:38,058 DEBUG [StoreCloser-TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/4d46d96b97ff4ec8b010199c78ab2aeb to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/4d46d96b97ff4ec8b010199c78ab2aeb 2023-04-19 20:55:38,059 DEBUG [StoreCloser-TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/c062767b9d634d2fbb9737f52c20e15a to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/c062767b9d634d2fbb9737f52c20e15a 2023-04-19 20:55:38,059 DEBUG [StoreCloser-TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/c8a8147791b546ac8b0cf5eb79a7cc66 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/c8a8147791b546ac8b0cf5eb79a7cc66 2023-04-19 20:55:38,060 DEBUG [StoreCloser-TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/79df7df7265d446dbcc5096b9e27a89e to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/79df7df7265d446dbcc5096b9e27a89e 2023-04-19 20:55:38,061 DEBUG [StoreCloser-TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/f354c78a24c445fd9b41bc8987f232d8 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/f354c78a24c445fd9b41bc8987f232d8 2023-04-19 20:55:38,062 DEBUG [StoreCloser-TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/8133d03912154df89d6084ed7259cdcc to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/8133d03912154df89d6084ed7259cdcc 2023-04-19 20:55:38,062 DEBUG [StoreCloser-TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/98ee35ec3c42466faf36f0d83c221698 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/98ee35ec3c42466faf36f0d83c221698 2023-04-19 20:55:38,063 DEBUG [StoreCloser-TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/a373fc3f0b68496894e5b98c7b87c4a2 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/a373fc3f0b68496894e5b98c7b87c4a2 2023-04-19 20:55:38,064 DEBUG [StoreCloser-TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/3e5713ebc792425caf9af7d5349a2857 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/3e5713ebc792425caf9af7d5349a2857 2023-04-19 20:55:38,064 DEBUG [StoreCloser-TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/ce774494e7894fbab682540dbcbb5957 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/ce774494e7894fbab682540dbcbb5957 2023-04-19 20:55:38,065 DEBUG [StoreCloser-TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/c2e0e314b81742009f61561aade9f5c4 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/c2e0e314b81742009f61561aade9f5c4 2023-04-19 20:55:38,066 DEBUG [StoreCloser-TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/c0d0d3c6ca2a463e9091e8ea9a542b4f to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/c0d0d3c6ca2a463e9091e8ea9a542b4f 2023-04-19 20:55:38,067 DEBUG [StoreCloser-TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/39089e48a7004a539f5f6177094991ef to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/39089e48a7004a539f5f6177094991ef 2023-04-19 20:55:38,067 DEBUG [StoreCloser-TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/5586d5e2347a45ee9e8ca97d97422563 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/5586d5e2347a45ee9e8ca97d97422563 2023-04-19 20:55:38,068 DEBUG [StoreCloser-TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/667c47b0a50540b7afef7896f3bc1e42 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/667c47b0a50540b7afef7896f3bc1e42 2023-04-19 20:55:38,069 DEBUG [StoreCloser-TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/0c5836c6cce74b26a94c5ea844769ba5 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/0c5836c6cce74b26a94c5ea844769ba5 2023-04-19 20:55:38,070 DEBUG [StoreCloser-TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/d1a42b823b8f4dd89fb9892c654e087d to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/d1a42b823b8f4dd89fb9892c654e087d 2023-04-19 20:55:38,071 DEBUG [StoreCloser-TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/64e8bbbcd36c465eb635f45f88c903ef to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/64e8bbbcd36c465eb635f45f88c903ef 2023-04-19 20:55:38,071 DEBUG [StoreCloser-TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/630b0bf052924e7a912e4e926138960b to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/630b0bf052924e7a912e4e926138960b 2023-04-19 20:55:38,072 DEBUG [StoreCloser-TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/b34d9975f0414ba4936e0ca44220489c to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/b34d9975f0414ba4936e0ca44220489c 2023-04-19 20:55:38,073 DEBUG [StoreCloser-TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/98f7d974b11c40d39d0310fa791d707a to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/98f7d974b11c40d39d0310fa791d707a 2023-04-19 20:55:38,073 DEBUG [StoreCloser-TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/45b5898ecfa84bf18388ed2e844643a2 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/45b5898ecfa84bf18388ed2e844643a2 2023-04-19 20:55:38,074 DEBUG [StoreCloser-TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/6ac196ad591443668c5308f0fd40bf62 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/6ac196ad591443668c5308f0fd40bf62 2023-04-19 20:55:38,077 DEBUG [StoreCloser-TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/40a475b1e1ee4d6485edc6590c3764c2, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/714fd4c7d8e243a39c0c8f83118f4ebe, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/f9c06ee3de7043fb945995fb46b3a40f, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/c94dccc5398c482ca67248b6814adb8e, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/0de8ee6282ed47b4b49617d785bca139, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/2c76ce6f46344485814067d8447f3a66, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/a807b8737638476cb03aceded8080cc7, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/bb55aa0fba75465fb625cbe11c6eadee, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/81030fc498f446279cd21a78db2b4868, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/71243cd9934b49af886a82cb5d787d7b, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/1fabe96869f7467aa6c4c29409fb6345, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/842f3fe790614916a9dfc6c04c3007ad, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/af1bd3ce6e9345748004cbb754a4efe6, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/e97ade3ae4cd43618db723d1870ca093, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/b0f6c9b9d2b04097bb7cfd9eac915eb8, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/b6c293283b58443fb4d9d7f739c6605c, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/8d37b8d547204ce9adb826423c855671, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/cb93da1182c9466498c985a1e5403163, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/3df8315ed49142128a18327ccd4909ca, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/5e1d15b820cb4c7abd7c39e6d81bff29, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/76d4439dacdd48548706a4feb1970175, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/9014be03cccd466cb79e8f76ebe0663f, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/35bf50208f7e426b8f03120b7923cbb1, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/16a796def9844ef086503cfc1ac05681, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/3da7f0565e994c93b116ef6abfbfe4ab, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/e5bdf32d0342460eb4fb5bc4e2569b37] to archive 2023-04-19 20:55:38,077 DEBUG [StoreCloser-TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-19 20:55:38,078 DEBUG [StoreCloser-TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/40a475b1e1ee4d6485edc6590c3764c2 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/40a475b1e1ee4d6485edc6590c3764c2 2023-04-19 20:55:38,079 DEBUG [StoreCloser-TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/714fd4c7d8e243a39c0c8f83118f4ebe to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/714fd4c7d8e243a39c0c8f83118f4ebe 2023-04-19 20:55:38,080 DEBUG [StoreCloser-TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/f9c06ee3de7043fb945995fb46b3a40f to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/f9c06ee3de7043fb945995fb46b3a40f 2023-04-19 20:55:38,081 DEBUG [StoreCloser-TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/c94dccc5398c482ca67248b6814adb8e to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/c94dccc5398c482ca67248b6814adb8e 2023-04-19 20:55:38,082 DEBUG [StoreCloser-TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/0de8ee6282ed47b4b49617d785bca139 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/0de8ee6282ed47b4b49617d785bca139 2023-04-19 20:55:38,082 DEBUG [StoreCloser-TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/2c76ce6f46344485814067d8447f3a66 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/2c76ce6f46344485814067d8447f3a66 2023-04-19 20:55:38,083 DEBUG [StoreCloser-TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/a807b8737638476cb03aceded8080cc7 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/a807b8737638476cb03aceded8080cc7 2023-04-19 20:55:38,084 DEBUG [StoreCloser-TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/bb55aa0fba75465fb625cbe11c6eadee to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/bb55aa0fba75465fb625cbe11c6eadee 2023-04-19 20:55:38,084 DEBUG [StoreCloser-TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/81030fc498f446279cd21a78db2b4868 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/81030fc498f446279cd21a78db2b4868 2023-04-19 20:55:38,085 DEBUG [StoreCloser-TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/71243cd9934b49af886a82cb5d787d7b to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/71243cd9934b49af886a82cb5d787d7b 2023-04-19 20:55:38,086 DEBUG [StoreCloser-TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/1fabe96869f7467aa6c4c29409fb6345 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/1fabe96869f7467aa6c4c29409fb6345 2023-04-19 20:55:38,087 DEBUG [StoreCloser-TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/842f3fe790614916a9dfc6c04c3007ad to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/842f3fe790614916a9dfc6c04c3007ad 2023-04-19 20:55:38,087 DEBUG [StoreCloser-TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/af1bd3ce6e9345748004cbb754a4efe6 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/af1bd3ce6e9345748004cbb754a4efe6 2023-04-19 20:55:38,088 DEBUG [StoreCloser-TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/e97ade3ae4cd43618db723d1870ca093 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/e97ade3ae4cd43618db723d1870ca093 2023-04-19 20:55:38,089 DEBUG [StoreCloser-TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/b0f6c9b9d2b04097bb7cfd9eac915eb8 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/b0f6c9b9d2b04097bb7cfd9eac915eb8 2023-04-19 20:55:38,090 DEBUG [StoreCloser-TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/b6c293283b58443fb4d9d7f739c6605c to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/b6c293283b58443fb4d9d7f739c6605c 2023-04-19 20:55:38,091 DEBUG [StoreCloser-TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/8d37b8d547204ce9adb826423c855671 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/8d37b8d547204ce9adb826423c855671 2023-04-19 20:55:38,091 DEBUG [StoreCloser-TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/cb93da1182c9466498c985a1e5403163 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/cb93da1182c9466498c985a1e5403163 2023-04-19 20:55:38,092 DEBUG [StoreCloser-TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/3df8315ed49142128a18327ccd4909ca to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/3df8315ed49142128a18327ccd4909ca 2023-04-19 20:55:38,093 DEBUG [StoreCloser-TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/5e1d15b820cb4c7abd7c39e6d81bff29 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/5e1d15b820cb4c7abd7c39e6d81bff29 2023-04-19 20:55:38,094 DEBUG [StoreCloser-TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/76d4439dacdd48548706a4feb1970175 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/76d4439dacdd48548706a4feb1970175 2023-04-19 20:55:38,094 DEBUG [StoreCloser-TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/9014be03cccd466cb79e8f76ebe0663f to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/9014be03cccd466cb79e8f76ebe0663f 2023-04-19 20:55:38,095 DEBUG [StoreCloser-TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/35bf50208f7e426b8f03120b7923cbb1 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/35bf50208f7e426b8f03120b7923cbb1 2023-04-19 20:55:38,096 DEBUG [StoreCloser-TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/16a796def9844ef086503cfc1ac05681 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/16a796def9844ef086503cfc1ac05681 2023-04-19 20:55:38,096 DEBUG [StoreCloser-TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/3da7f0565e994c93b116ef6abfbfe4ab to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/3da7f0565e994c93b116ef6abfbfe4ab 2023-04-19 20:55:38,097 DEBUG [StoreCloser-TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/e5bdf32d0342460eb4fb5bc4e2569b37 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/e5bdf32d0342460eb4fb5bc4e2569b37 2023-04-19 20:55:38,100 DEBUG [StoreCloser-TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/2c6297cece2044c4bae3dd80658097ed, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/bc27b0c6aec54ca2b7e4ddf1011d33db, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/6d9c33d398a24bbcb7c55c1ee71e29ef, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/8e7ba01953844fb8951f5c20644aee91, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/977c283471774b328ccb52d7451ffa12, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/adaf97591f734ec69d8eb866392635b5, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/4be5c3b40ceb4154a4579a3303b110a8, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/b0b7f37b35aa4eb0847fa997dd7e3d43, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/24849ed7896946869e46a41c455c5828, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/88399863f67946f4b492e08c77381c49, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/8ec186b6992642ffaafc818130f98183, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/7ffd3c4c8f9a4b26b30d9ff4b109fcca, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/68037ac8c97b40a4be643b017208beeb, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/4d1c174328d4411db83ab3f03b088421, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/2bd2594f12a24cb2a90bc65be23be928, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/837e73bff1614b42b3e583dd9d6c6d0a, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/dd9ebbf18d3d4a68a1b87918fcd26765, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/514bc54edae5475496f0bc331b053218, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/611c5cfb2cba471e96fb35643f3fc14d, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/18084385e97d480d9ac6b49db46c2744, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/b01ff249fa494258bb916bd2bbb7d275, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/b631298e59a745c79915353c9094e307, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/2218ea2f6bfc45dd9ef51b2241596fbc, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/f3ca9c30a1184edab9fa8fcc03fa5092, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/67ce81fa975648bf9dcb10a4c009ce00, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/21265545ee324f04a4496c82b88454bd] to archive 2023-04-19 20:55:38,100 DEBUG [StoreCloser-TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-19 20:55:38,102 DEBUG [StoreCloser-TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/2c6297cece2044c4bae3dd80658097ed to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/2c6297cece2044c4bae3dd80658097ed 2023-04-19 20:55:38,103 DEBUG [StoreCloser-TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/bc27b0c6aec54ca2b7e4ddf1011d33db to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/bc27b0c6aec54ca2b7e4ddf1011d33db 2023-04-19 20:55:38,104 DEBUG [StoreCloser-TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/6d9c33d398a24bbcb7c55c1ee71e29ef to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/6d9c33d398a24bbcb7c55c1ee71e29ef 2023-04-19 20:55:38,105 DEBUG [StoreCloser-TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/8e7ba01953844fb8951f5c20644aee91 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/8e7ba01953844fb8951f5c20644aee91 2023-04-19 20:55:38,106 DEBUG [StoreCloser-TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/977c283471774b328ccb52d7451ffa12 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/977c283471774b328ccb52d7451ffa12 2023-04-19 20:55:38,107 DEBUG [StoreCloser-TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/adaf97591f734ec69d8eb866392635b5 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/adaf97591f734ec69d8eb866392635b5 2023-04-19 20:55:38,108 DEBUG [StoreCloser-TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/4be5c3b40ceb4154a4579a3303b110a8 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/4be5c3b40ceb4154a4579a3303b110a8 2023-04-19 20:55:38,109 DEBUG [StoreCloser-TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/b0b7f37b35aa4eb0847fa997dd7e3d43 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/b0b7f37b35aa4eb0847fa997dd7e3d43 2023-04-19 20:55:38,112 DEBUG [StoreCloser-TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/24849ed7896946869e46a41c455c5828 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/24849ed7896946869e46a41c455c5828 2023-04-19 20:55:38,114 DEBUG [StoreCloser-TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/88399863f67946f4b492e08c77381c49 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/88399863f67946f4b492e08c77381c49 2023-04-19 20:55:38,115 DEBUG [StoreCloser-TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/8ec186b6992642ffaafc818130f98183 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/8ec186b6992642ffaafc818130f98183 2023-04-19 20:55:38,116 DEBUG [StoreCloser-TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/7ffd3c4c8f9a4b26b30d9ff4b109fcca to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/7ffd3c4c8f9a4b26b30d9ff4b109fcca 2023-04-19 20:55:38,117 DEBUG [StoreCloser-TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/68037ac8c97b40a4be643b017208beeb to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/68037ac8c97b40a4be643b017208beeb 2023-04-19 20:55:38,118 DEBUG [StoreCloser-TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/4d1c174328d4411db83ab3f03b088421 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/4d1c174328d4411db83ab3f03b088421 2023-04-19 20:55:38,119 DEBUG [StoreCloser-TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/2bd2594f12a24cb2a90bc65be23be928 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/2bd2594f12a24cb2a90bc65be23be928 2023-04-19 20:55:38,120 DEBUG [StoreCloser-TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/837e73bff1614b42b3e583dd9d6c6d0a to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/837e73bff1614b42b3e583dd9d6c6d0a 2023-04-19 20:55:38,121 DEBUG [StoreCloser-TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/dd9ebbf18d3d4a68a1b87918fcd26765 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/dd9ebbf18d3d4a68a1b87918fcd26765 2023-04-19 20:55:38,122 DEBUG [StoreCloser-TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/514bc54edae5475496f0bc331b053218 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/514bc54edae5475496f0bc331b053218 2023-04-19 20:55:38,123 DEBUG [StoreCloser-TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/611c5cfb2cba471e96fb35643f3fc14d to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/611c5cfb2cba471e96fb35643f3fc14d 2023-04-19 20:55:38,124 DEBUG [StoreCloser-TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/18084385e97d480d9ac6b49db46c2744 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/18084385e97d480d9ac6b49db46c2744 2023-04-19 20:55:38,125 DEBUG [StoreCloser-TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/b01ff249fa494258bb916bd2bbb7d275 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/b01ff249fa494258bb916bd2bbb7d275 2023-04-19 20:55:38,126 DEBUG [StoreCloser-TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/b631298e59a745c79915353c9094e307 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/b631298e59a745c79915353c9094e307 2023-04-19 20:55:38,127 DEBUG [StoreCloser-TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/2218ea2f6bfc45dd9ef51b2241596fbc to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/2218ea2f6bfc45dd9ef51b2241596fbc 2023-04-19 20:55:38,128 DEBUG [StoreCloser-TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/f3ca9c30a1184edab9fa8fcc03fa5092 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/f3ca9c30a1184edab9fa8fcc03fa5092 2023-04-19 20:55:38,129 DEBUG [StoreCloser-TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/67ce81fa975648bf9dcb10a4c009ce00 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/67ce81fa975648bf9dcb10a4c009ce00 2023-04-19 20:55:38,130 DEBUG [StoreCloser-TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/21265545ee324f04a4496c82b88454bd to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/21265545ee324f04a4496c82b88454bd 2023-04-19 20:55:38,136 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/recovered.edits/1017.seqid, newMaxSeqId=1017, maxSeqId=4 2023-04-19 20:55:38,138 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc. 2023-04-19 20:55:38,138 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 5067d98ca5f8d3760c5d0eb2b8bb06dc: 2023-04-19 20:55:38,139 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 5067d98ca5f8d3760c5d0eb2b8bb06dc 2023-04-19 20:55:38,140 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=44 updating hbase:meta row=5067d98ca5f8d3760c5d0eb2b8bb06dc, regionState=CLOSED 2023-04-19 20:55:38,140 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681937738140"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681937738140"}]},"ts":"1681937738140"} 2023-04-19 20:55:38,142 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=45, resume processing ppid=44 2023-04-19 20:55:38,142 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=45, ppid=44, state=SUCCESS; CloseRegionProcedure 5067d98ca5f8d3760c5d0eb2b8bb06dc, server=jenkins-hbase4.apache.org,35247,1681937616971 in 3.2480 sec 2023-04-19 20:55:38,144 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=44, resume processing ppid=43 2023-04-19 20:55:38,144 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=44, ppid=43, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=5067d98ca5f8d3760c5d0eb2b8bb06dc, UNASSIGN in 3.2520 sec 2023-04-19 20:55:38,144 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681937738144"}]},"ts":"1681937738144"} 2023-04-19 20:55:38,145 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-19 20:55:38,146 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-19 20:55:38,148 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=43, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 3.3640 sec 2023-04-19 20:55:38,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-19 20:55:38,893 INFO [Listener at localhost/34893] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 43 completed 2023-04-19 20:55:38,894 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-19 20:55:38,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure2.ProcedureExecutor(1029): Stored pid=46, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-19 20:55:38,896 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=46, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-19 20:55:38,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1227): Checking to see if procedure is done pid=46 2023-04-19 20:55:38,896 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=46, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-19 20:55:38,900 DEBUG [HFileArchiver-9] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/.tmp/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc 2023-04-19 20:55:38,901 DEBUG [HFileArchiver-9] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/.tmp/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A, FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/.tmp/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B, FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/.tmp/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C, FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/.tmp/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/recovered.edits] 2023-04-19 20:55:38,904 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/.tmp/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/c6730b93fc0442728faf7d36918c6231 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/c6730b93fc0442728faf7d36918c6231 2023-04-19 20:55:38,905 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/.tmp/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/dd27c1508d1e4210b9b89b0ddac58857 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/dd27c1508d1e4210b9b89b0ddac58857 2023-04-19 20:55:38,906 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/.tmp/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/df7681c8951b43f5882299ca662ceb3a to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/A/df7681c8951b43f5882299ca662ceb3a 2023-04-19 20:55:38,908 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/.tmp/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/a7f6696a3f77400284473332299b9843 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/a7f6696a3f77400284473332299b9843 2023-04-19 20:55:38,909 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/.tmp/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/b8bb0dcf88e1466a92d87da018aab604 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/b8bb0dcf88e1466a92d87da018aab604 2023-04-19 20:55:38,911 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/.tmp/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/cd30f732e95843f7ada8bdd95136580f to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/B/cd30f732e95843f7ada8bdd95136580f 2023-04-19 20:55:38,913 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/.tmp/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/44a93841c8484e7a91f293cf83b2d0c1 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/44a93841c8484e7a91f293cf83b2d0c1 2023-04-19 20:55:38,914 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/.tmp/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/d42cfa1dc0614d05a925426ce09cc4e7 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/d42cfa1dc0614d05a925426ce09cc4e7 2023-04-19 20:55:38,916 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/.tmp/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/e7b3f3d4403148cc974610ff1780896c to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/C/e7b3f3d4403148cc974610ff1780896c 2023-04-19 20:55:38,918 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/.tmp/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/recovered.edits/1017.seqid to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc/recovered.edits/1017.seqid 2023-04-19 20:55:38,919 DEBUG [HFileArchiver-9] backup.HFileArchiver(596): Deleted hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/.tmp/data/default/TestAcidGuarantees/5067d98ca5f8d3760c5d0eb2b8bb06dc 2023-04-19 20:55:38,919 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-19 20:55:38,919 DEBUG [PEWorker-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-19 20:55:38,920 DEBUG [PEWorker-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-04-19 20:55:38,922 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304190452a716afcf45499fa9aa9bc0846380 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304190452a716afcf45499fa9aa9bc0846380 2023-04-19 20:55:38,923 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419099208ffcc52453da732f4af1e8c9166 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419099208ffcc52453da732f4af1e8c9166 2023-04-19 20:55:38,925 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041914b132754ce14bffac33c7e4d6f46007 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041914b132754ce14bffac33c7e4d6f46007 2023-04-19 20:55:38,926 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041919499162944d4af98a6bd89a717f1bcd to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041919499162944d4af98a6bd89a717f1bcd 2023-04-19 20:55:38,927 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304191cbe62fdaba54a37a466b90e837be380 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304191cbe62fdaba54a37a466b90e837be380 2023-04-19 20:55:38,928 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304192acb5515b7de4bb1a2483d65f7a2e563 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304192acb5515b7de4bb1a2483d65f7a2e563 2023-04-19 20:55:38,929 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304192b2fe1cc8fc341edae1d0b8db7a5f0eb to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304192b2fe1cc8fc341edae1d0b8db7a5f0eb 2023-04-19 20:55:38,930 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419484d94f6ed974ad9a921ea43a9f467f3 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419484d94f6ed974ad9a921ea43a9f467f3 2023-04-19 20:55:38,931 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419781ad46d1e924141b8b426b70b51d8ca to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419781ad46d1e924141b8b426b70b51d8ca 2023-04-19 20:55:38,933 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419825ebeb9a334448f831c4649acd54178 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419825ebeb9a334448f831c4649acd54178 2023-04-19 20:55:38,934 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419832223214dac47ebb7e4b6713b599801 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419832223214dac47ebb7e4b6713b599801 2023-04-19 20:55:38,935 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304198966459f2a984c91859c4a203290cac4 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304198966459f2a984c91859c4a203290cac4 2023-04-19 20:55:38,936 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304198e2c5f7ae35f4aa68fa98debd1d7cca9 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304198e2c5f7ae35f4aa68fa98debd1d7cca9 2023-04-19 20:55:38,937 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041992b89899a1184268bad92eb8fb38621e to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041992b89899a1184268bad92eb8fb38621e 2023-04-19 20:55:38,939 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041998988b2aed9549308ee8974217c9f865 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041998988b2aed9549308ee8974217c9f865 2023-04-19 20:55:38,940 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304199fdabb2747684160b915f4905b1d89f8 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304199fdabb2747684160b915f4905b1d89f8 2023-04-19 20:55:38,942 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419b3fdc6cbed934313b41ec3bbe1fee13f to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419b3fdc6cbed934313b41ec3bbe1fee13f 2023-04-19 20:55:38,943 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419b71ef916b00c46028e9f9db7eebb90a3 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419b71ef916b00c46028e9f9db7eebb90a3 2023-04-19 20:55:38,944 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419c232f264476c48b486bbc3a34c60e1cc to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419c232f264476c48b486bbc3a34c60e1cc 2023-04-19 20:55:38,946 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419d20601e9137540faaf4d3f389963337f to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419d20601e9137540faaf4d3f389963337f 2023-04-19 20:55:38,947 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419e8b4ed14982541618d6cbe67c38810c0 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419e8b4ed14982541618d6cbe67c38810c0 2023-04-19 20:55:38,948 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419edf0c10320be40c3a4f09a94b81e613f to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419edf0c10320be40c3a4f09a94b81e613f 2023-04-19 20:55:38,949 DEBUG [PEWorker-5] backup.HFileArchiver(596): Deleted hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-19 20:55:38,951 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=46, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-19 20:55:38,952 WARN [PEWorker-5] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-19 20:55:38,954 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-19 20:55:38,954 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=46, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-19 20:55:38,955 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-19 20:55:38,955 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1681937738955"}]},"ts":"9223372036854775807"} 2023-04-19 20:55:38,957 INFO [PEWorker-5] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-19 20:55:38,957 DEBUG [PEWorker-5] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 5067d98ca5f8d3760c5d0eb2b8bb06dc, NAME => 'TestAcidGuarantees,,1681937710504.5067d98ca5f8d3760c5d0eb2b8bb06dc.', STARTKEY => '', ENDKEY => ''}] 2023-04-19 20:55:38,957 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-19 20:55:38,957 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1681937738957"}]},"ts":"9223372036854775807"} 2023-04-19 20:55:38,958 INFO [PEWorker-5] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-19 20:55:38,960 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(130): Finished pid=46, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-19 20:55:38,961 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=46, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 66 msec 2023-04-19 20:55:38,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1227): Checking to see if procedure is done pid=46 2023-04-19 20:55:38,997 INFO [Listener at localhost/34893] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 46 completed 2023-04-19 20:55:39,010 INFO [Listener at localhost/34893] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobScanAtomicity Thread=228 (was 228), OpenFileDescriptor=561 (was 557) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=394 (was 331) - SystemLoadAverage LEAK? -, ProcessCount=172 (was 172), AvailableMemoryMB=6275 (was 5509) - AvailableMemoryMB LEAK? - 2023-04-19 20:55:39,021 INFO [Listener at localhost/34893] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testScanAtomicity Thread=228, OpenFileDescriptor=561, MaxFileDescriptor=60000, SystemLoadAverage=394, ProcessCount=172, AvailableMemoryMB=6274 2023-04-19 20:55:39,023 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-19 20:55:39,024 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-19 20:55:39,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure2.ProcedureExecutor(1029): Stored pid=47, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-19 20:55:39,027 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-19 20:55:39,027 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 47 2023-04-19 20:55:39,027 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-19 20:55:39,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-19 20:55:39,029 DEBUG [HFileArchiver-10] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/.tmp/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d 2023-04-19 20:55:39,029 DEBUG [HFileArchiver-10] backup.HFileArchiver(153): Directory hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/.tmp/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d empty. 2023-04-19 20:55:39,030 DEBUG [HFileArchiver-10] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/.tmp/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d 2023-04-19 20:55:39,030 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-19 20:55:39,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-19 20:55:39,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-19 20:55:39,437 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-19 20:55:39,439 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 4393734ae917e967cf2b5b4b3b44ed0d, NAME => 'TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/.tmp 2023-04-19 20:55:39,462 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-19 20:55:39,462 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 4393734ae917e967cf2b5b4b3b44ed0d, disabling compactions & flushes 2023-04-19 20:55:39,462 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:39,462 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:39,462 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. after waiting 0 ms 2023-04-19 20:55:39,462 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:39,462 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:39,462 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 4393734ae917e967cf2b5b4b3b44ed0d: 2023-04-19 20:55:39,464 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-19 20:55:39,465 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681937739465"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681937739465"}]},"ts":"1681937739465"} 2023-04-19 20:55:39,466 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-19 20:55:39,467 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-19 20:55:39,467 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681937739467"}]},"ts":"1681937739467"} 2023-04-19 20:55:39,468 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-19 20:55:39,471 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=4393734ae917e967cf2b5b4b3b44ed0d, ASSIGN}] 2023-04-19 20:55:39,473 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=4393734ae917e967cf2b5b4b3b44ed0d, ASSIGN 2023-04-19 20:55:39,473 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=4393734ae917e967cf2b5b4b3b44ed0d, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,35247,1681937616971; forceNewPlan=false, retain=false 2023-04-19 20:55:39,625 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=48 updating hbase:meta row=4393734ae917e967cf2b5b4b3b44ed0d, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:39,625 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681937739625"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681937739625"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681937739625"}]},"ts":"1681937739625"} 2023-04-19 20:55:39,627 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=49, ppid=48, state=RUNNABLE; OpenRegionProcedure 4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971}] 2023-04-19 20:55:39,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-19 20:55:39,781 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:39,782 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 4393734ae917e967cf2b5b4b3b44ed0d, NAME => 'TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.', STARTKEY => '', ENDKEY => ''} 2023-04-19 20:55:39,782 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 4393734ae917e967cf2b5b4b3b44ed0d 2023-04-19 20:55:39,782 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-19 20:55:39,782 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 4393734ae917e967cf2b5b4b3b44ed0d 2023-04-19 20:55:39,782 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 4393734ae917e967cf2b5b4b3b44ed0d 2023-04-19 20:55:39,783 INFO [StoreOpener-4393734ae917e967cf2b5b4b3b44ed0d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 4393734ae917e967cf2b5b4b3b44ed0d 2023-04-19 20:55:39,785 DEBUG [StoreOpener-4393734ae917e967cf2b5b4b3b44ed0d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A 2023-04-19 20:55:39,785 DEBUG [StoreOpener-4393734ae917e967cf2b5b4b3b44ed0d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A 2023-04-19 20:55:39,785 INFO [StoreOpener-4393734ae917e967cf2b5b4b3b44ed0d-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-19 20:55:39,785 INFO [StoreOpener-4393734ae917e967cf2b5b4b3b44ed0d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 4393734ae917e967cf2b5b4b3b44ed0d columnFamilyName A 2023-04-19 20:55:39,786 INFO [StoreOpener-4393734ae917e967cf2b5b4b3b44ed0d-1] regionserver.HStore(310): Store=4393734ae917e967cf2b5b4b3b44ed0d/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-19 20:55:39,786 INFO [StoreOpener-4393734ae917e967cf2b5b4b3b44ed0d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 4393734ae917e967cf2b5b4b3b44ed0d 2023-04-19 20:55:39,787 DEBUG [StoreOpener-4393734ae917e967cf2b5b4b3b44ed0d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B 2023-04-19 20:55:39,787 DEBUG [StoreOpener-4393734ae917e967cf2b5b4b3b44ed0d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B 2023-04-19 20:55:39,787 INFO [StoreOpener-4393734ae917e967cf2b5b4b3b44ed0d-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-19 20:55:39,787 INFO [StoreOpener-4393734ae917e967cf2b5b4b3b44ed0d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 4393734ae917e967cf2b5b4b3b44ed0d columnFamilyName B 2023-04-19 20:55:39,788 INFO [StoreOpener-4393734ae917e967cf2b5b4b3b44ed0d-1] regionserver.HStore(310): Store=4393734ae917e967cf2b5b4b3b44ed0d/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-19 20:55:39,788 INFO [StoreOpener-4393734ae917e967cf2b5b4b3b44ed0d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 4393734ae917e967cf2b5b4b3b44ed0d 2023-04-19 20:55:39,789 DEBUG [StoreOpener-4393734ae917e967cf2b5b4b3b44ed0d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C 2023-04-19 20:55:39,789 DEBUG [StoreOpener-4393734ae917e967cf2b5b4b3b44ed0d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C 2023-04-19 20:55:39,789 INFO [StoreOpener-4393734ae917e967cf2b5b4b3b44ed0d-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-19 20:55:39,789 INFO [StoreOpener-4393734ae917e967cf2b5b4b3b44ed0d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 4393734ae917e967cf2b5b4b3b44ed0d columnFamilyName C 2023-04-19 20:55:39,790 INFO [StoreOpener-4393734ae917e967cf2b5b4b3b44ed0d-1] regionserver.HStore(310): Store=4393734ae917e967cf2b5b4b3b44ed0d/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-19 20:55:39,790 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:39,790 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d 2023-04-19 20:55:39,791 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d 2023-04-19 20:55:39,792 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-19 20:55:39,793 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 4393734ae917e967cf2b5b4b3b44ed0d 2023-04-19 20:55:39,794 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-19 20:55:39,795 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 4393734ae917e967cf2b5b4b3b44ed0d; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=74891707, jitterRate=0.11597339808940887}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-19 20:55:39,795 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 4393734ae917e967cf2b5b4b3b44ed0d: 2023-04-19 20:55:39,795 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d., pid=49, masterSystemTime=1681937739778 2023-04-19 20:55:39,797 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:39,797 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:39,797 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=48 updating hbase:meta row=4393734ae917e967cf2b5b4b3b44ed0d, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:39,797 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681937739797"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681937739797"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681937739797"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681937739797"}]},"ts":"1681937739797"} 2023-04-19 20:55:39,799 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=49, resume processing ppid=48 2023-04-19 20:55:39,799 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=49, ppid=48, state=SUCCESS; OpenRegionProcedure 4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 in 172 msec 2023-04-19 20:55:39,800 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=48, resume processing ppid=47 2023-04-19 20:55:39,800 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=48, ppid=47, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=4393734ae917e967cf2b5b4b3b44ed0d, ASSIGN in 328 msec 2023-04-19 20:55:39,801 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-19 20:55:39,801 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681937739801"}]},"ts":"1681937739801"} 2023-04-19 20:55:39,802 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-19 20:55:39,808 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-19 20:55:39,809 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=47, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 784 msec 2023-04-19 20:55:40,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-19 20:55:40,132 INFO [Listener at localhost/34893] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 47 completed 2023-04-19 20:55:40,132 DEBUG [Listener at localhost/34893] zookeeper.ReadOnlyZKClient(139): Connect 0x66369677 to 127.0.0.1:58736 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-19 20:55:40,137 DEBUG [Listener at localhost/34893] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1df28b68, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-19 20:55:40,138 DEBUG [Listener at localhost/34893] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-19 20:55:40,141 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:35458, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-19 20:55:40,142 DEBUG [Listener at localhost/34893] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-19 20:55:40,143 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:57600, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-19 20:55:40,145 DEBUG [Listener at localhost/34893] zookeeper.ReadOnlyZKClient(139): Connect 0x6e0b6187 to 127.0.0.1:58736 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-19 20:55:40,149 DEBUG [Listener at localhost/34893] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3c5cd71f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-19 20:55:40,149 DEBUG [Listener at localhost/34893] zookeeper.ReadOnlyZKClient(139): Connect 0x2ab56e54 to 127.0.0.1:58736 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-19 20:55:40,159 DEBUG [Listener at localhost/34893] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1f0735dc, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-19 20:55:40,159 DEBUG [Listener at localhost/34893] zookeeper.ReadOnlyZKClient(139): Connect 0x15840296 to 127.0.0.1:58736 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-19 20:55:40,166 DEBUG [Listener at localhost/34893] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@14c209bc, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-19 20:55:40,167 DEBUG [Listener at localhost/34893] zookeeper.ReadOnlyZKClient(139): Connect 0x3816bdf5 to 127.0.0.1:58736 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-19 20:55:40,174 DEBUG [Listener at localhost/34893] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@db388d3, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-19 20:55:40,175 DEBUG [Listener at localhost/34893] zookeeper.ReadOnlyZKClient(139): Connect 0x50150525 to 127.0.0.1:58736 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-19 20:55:40,182 DEBUG [Listener at localhost/34893] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5de1e3c7, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-19 20:55:40,183 DEBUG [Listener at localhost/34893] zookeeper.ReadOnlyZKClient(139): Connect 0x7a4e3b2e to 127.0.0.1:58736 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-19 20:55:40,190 DEBUG [Listener at localhost/34893] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@28e3107b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-19 20:55:40,190 DEBUG [Listener at localhost/34893] zookeeper.ReadOnlyZKClient(139): Connect 0x1f65627a to 127.0.0.1:58736 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-19 20:55:40,200 DEBUG [Listener at localhost/34893] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@14f9adf3, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-19 20:55:40,200 DEBUG [Listener at localhost/34893] zookeeper.ReadOnlyZKClient(139): Connect 0x15f74c78 to 127.0.0.1:58736 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-19 20:55:40,207 DEBUG [Listener at localhost/34893] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@21359a0e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-19 20:55:40,208 DEBUG [Listener at localhost/34893] zookeeper.ReadOnlyZKClient(139): Connect 0x1401cac1 to 127.0.0.1:58736 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-19 20:55:40,219 DEBUG [Listener at localhost/34893] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3bbac0b5, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-19 20:55:40,219 DEBUG [Listener at localhost/34893] zookeeper.ReadOnlyZKClient(139): Connect 0x1f6e865f to 127.0.0.1:58736 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-19 20:55:40,231 DEBUG [Listener at localhost/34893] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5a300b1, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-19 20:55:40,246 DEBUG [hconnection-0x2a61cedf-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-19 20:55:40,262 DEBUG [hconnection-0x60d24e0b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-19 20:55:40,262 DEBUG [hconnection-0x279d72ae-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-19 20:55:40,262 DEBUG [hconnection-0x71c94fcc-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-19 20:55:40,263 DEBUG [hconnection-0x43cc87cd-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-19 20:55:40,263 DEBUG [hconnection-0x3197019a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-19 20:55:40,264 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:35484, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-19 20:55:40,264 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:35490, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-19 20:55:40,265 DEBUG [hconnection-0x46731fe2-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-19 20:55:40,268 DEBUG [hconnection-0x697c440e-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-19 20:55:40,268 DEBUG [hconnection-0x35e81b4f-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-19 20:55:40,269 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:35516, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-19 20:55:40,284 DEBUG [hconnection-0x75fbf211-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-19 20:55:40,284 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:35474, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-19 20:55:40,284 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:35492, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-19 20:55:40,285 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:35520, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-19 20:55:40,289 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:35470, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-19 20:55:40,289 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:40,289 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:35486, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-19 20:55:40,290 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:35504, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-19 20:55:40,290 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:35522, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-19 20:55:40,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:40,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:40,300 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:40,300 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:40,300 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:40,300 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:40,300 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:40,301 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:40,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(9158): Flush requested on 4393734ae917e967cf2b5b4b3b44ed0d 2023-04-19 20:55:40,309 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 4393734ae917e967cf2b5b4b3b44ed0d 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-19 20:55:40,310 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:40,310 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:40,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:40,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:40,311 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:40,311 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:40,311 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4393734ae917e967cf2b5b4b3b44ed0d, store=A 2023-04-19 20:55:40,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:40,311 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:55:40,311 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4393734ae917e967cf2b5b4b3b44ed0d, store=B 2023-04-19 20:55:40,311 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:55:40,311 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4393734ae917e967cf2b5b4b3b44ed0d, store=C 2023-04-19 20:55:40,311 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:55:40,312 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:40,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:40,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:40,312 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:40,332 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:40,332 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:40,332 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:40,332 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:40,335 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:40,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35492 deadline: 1681937800334, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:40,336 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:40,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 5 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35486 deadline: 1681937800334, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:40,336 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:40,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35490 deadline: 1681937800335, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:40,340 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:40,340 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:40,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35470 deadline: 1681937800340, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:40,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35474 deadline: 1681937800340, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:40,355 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:40,355 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:40,355 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:40,355 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. started... 2023-04-19 20:55:40,356 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. as already flushing 2023-04-19 20:55:40,356 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:40,356 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:40,356 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:40,356 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:40,364 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:40,364 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:40,370 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:40,370 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:40,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:40,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:40,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:40,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:40,371 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:40,371 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:40,371 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=15 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/A/593cad6360284a51a3288515b09116e0 2023-04-19 20:55:40,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:40,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:40,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:40,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:40,373 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:40,373 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7edb3ccf[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:40,373 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:40,373 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:40,375 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:40,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:40,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:40,375 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:40,375 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:40,375 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:40,375 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:40,375 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:40,378 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:40,378 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:40,378 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:40,378 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:40,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:40,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:40,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:40,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:40,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:40,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:40,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:40,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:40,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:40,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:40,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:40,382 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:40,382 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:40,382 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:40,382 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:40,382 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:40,382 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:40,384 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:40,384 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:40,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:40,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:40,384 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:40,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:40,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:40,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:40,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:40,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:40,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:40,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:40,387 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:40,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:40,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:40,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:40,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:40,388 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:40,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:40,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:40,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:40,392 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:40,392 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:40,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:40,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:40,392 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:40,392 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:40,392 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:40,392 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:40,392 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:40,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:40,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:40,393 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:40,393 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:40,392 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:40,392 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:40,393 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:40,393 DEBUG [Thread-3028] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:40,393 DEBUG [Thread-3028] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:40,437 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:40,437 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:40,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35486 deadline: 1681937800437, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:40,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35492 deadline: 1681937800437, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:40,438 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:40,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35490 deadline: 1681937800438, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:40,442 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:40,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35470 deadline: 1681937800442, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:40,443 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:40,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35474 deadline: 1681937800442, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:40,493 DEBUG [Thread-3028] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:40,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:40,497 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:40,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:40,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:40,503 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:40,504 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:40,504 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:40,504 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:40,504 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:40,504 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:40,512 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:40,512 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:40,513 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:40,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:40,513 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:40,513 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:40,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:40,513 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:40,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:40,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:40,514 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:40,522 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:55:40,522 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:40,522 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:40,522 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:40,523 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:40,527 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:40,527 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:40,527 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:40,527 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. started... 2023-04-19 20:55:40,527 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. as already flushing 2023-04-19 20:55:40,527 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:40,528 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:40,528 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:40,528 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:40,528 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:40,528 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:40,530 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:40,530 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:40,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:40,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:40,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:40,531 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:40,531 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:40,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:40,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:40,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:40,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:40,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:40,532 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:40,532 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2a863f7f[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:40,532 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:40,532 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:40,534 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:40,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:40,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:40,534 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:40,534 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:40,534 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:40,534 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:40,534 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:40,535 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:40,535 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:40,536 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:40,536 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:40,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:40,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:40,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:40,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:40,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:40,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:40,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:40,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:40,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:40,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:40,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:40,538 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:40,538 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:40,538 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:40,538 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:40,538 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:40,538 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:40,542 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:40,542 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:40,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:40,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:40,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:40,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:40,542 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:40,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:40,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:40,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:40,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:40,543 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:40,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:40,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:40,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:40,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:40,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:40,544 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:40,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:40,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:40,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:40,548 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:40,548 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:40,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:40,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:40,548 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:40,548 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:40,548 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:40,548 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:40,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:40,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:40,548 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:40,548 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:40,548 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:40,549 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:40,549 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:40,549 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:40,549 DEBUG [Thread-3028] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:40,549 DEBUG [Thread-3028] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:40,641 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:40,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35492 deadline: 1681937800638, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:40,642 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:40,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35486 deadline: 1681937800639, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:40,642 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:40,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35490 deadline: 1681937800639, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:40,645 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:40,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35474 deadline: 1681937800644, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:40,649 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:40,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35470 deadline: 1681937800649, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:40,649 DEBUG [Thread-3028] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:40,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:40,651 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:40,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:40,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:40,660 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:40,660 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:40,660 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:40,660 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:40,661 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:40,661 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:40,662 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:40,662 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:40,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:40,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:40,665 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:40,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:40,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:40,666 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:40,667 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:40,667 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:55:40,667 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:40,667 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:40,667 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:40,667 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:40,667 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:40,667 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:40,672 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:40,672 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:40,672 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:40,673 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. started... 2023-04-19 20:55:40,673 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. as already flushing 2023-04-19 20:55:40,673 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:40,673 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:40,673 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:40,673 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:40,681 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:40,681 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:40,683 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:40,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:40,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:40,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:40,683 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:40,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:40,683 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:40,684 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:40,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:40,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:40,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:40,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:40,685 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:40,685 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3a199b3c[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:40,685 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:40,685 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:40,686 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:40,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:40,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:40,686 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:40,686 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:40,686 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:40,687 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:40,687 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:40,689 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:40,689 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:40,689 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:40,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:40,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:40,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:40,689 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:40,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:40,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:40,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:40,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:40,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:40,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:40,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:40,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:40,691 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:40,691 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:40,691 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:40,691 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:40,692 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:40,692 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:40,695 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:40,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:40,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:40,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:40,695 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:40,695 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:40,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:40,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:40,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:40,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:40,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:40,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:40,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:40,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:40,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:40,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:40,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:40,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:40,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:40,699 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:40,700 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:40,702 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:40,702 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:40,702 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:40,702 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:40,702 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:40,703 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:40,703 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:40,703 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:40,703 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:40,703 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:40,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:40,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:40,703 DEBUG [Thread-3028] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:40,702 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:40,703 DEBUG [Thread-3028] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:40,703 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:40,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:40,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:40,803 DEBUG [Thread-3028] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:40,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:40,806 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:40,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:40,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:40,807 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:40,807 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:40,807 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:40,807 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:40,808 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:40,808 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:40,809 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:40,809 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:40,809 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:40,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:40,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:40,810 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:40,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:40,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:40,811 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:40,811 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:40,811 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:40,813 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:55:40,813 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:40,813 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:40,813 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:40,813 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:40,819 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:40,819 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:40,820 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:40,820 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. started... 2023-04-19 20:55:40,820 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. as already flushing 2023-04-19 20:55:40,820 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:40,820 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:40,820 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:40,820 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:40,820 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:40,820 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:40,823 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:40,823 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:40,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:40,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:40,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:40,824 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:40,824 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:40,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:40,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:40,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:40,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:40,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:40,825 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:40,825 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1395b2d8[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:40,825 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:40,825 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:40,827 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:40,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:40,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:40,827 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:40,827 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:40,827 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:40,827 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:40,827 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:40,830 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:40,830 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:40,830 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:40,830 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:40,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:40,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:40,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:40,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:40,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:40,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:40,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:40,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:40,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:40,833 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=15 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/B/c20e704eaa9047948b65bb0521962ae2 2023-04-19 20:55:40,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:40,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:40,834 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:40,834 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:40,834 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:40,834 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:40,834 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:40,834 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:40,837 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:40,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:40,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:40,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:40,837 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:40,838 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:40,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:40,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:40,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:40,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:40,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:40,838 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:40,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:40,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:40,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:40,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:40,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:40,840 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:40,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:40,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:40,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:40,844 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:40,844 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:40,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:40,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:40,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:40,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:40,845 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:40,845 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:40,845 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:40,846 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:40,846 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:40,846 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:40,846 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:40,846 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:40,845 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:40,847 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:40,847 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:55:40,847 DEBUG [Thread-3028] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:40,847 DEBUG [Thread-3028] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:40,883 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=15 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/C/b6ecd3d5a379435b990ecc11fae75469 2023-04-19 20:55:40,889 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/A/593cad6360284a51a3288515b09116e0 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/593cad6360284a51a3288515b09116e0 2023-04-19 20:55:40,896 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/593cad6360284a51a3288515b09116e0, entries=150, sequenceid=15, filesize=11.6 K 2023-04-19 20:55:40,897 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/B/c20e704eaa9047948b65bb0521962ae2 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/c20e704eaa9047948b65bb0521962ae2 2023-04-19 20:55:40,902 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/c20e704eaa9047948b65bb0521962ae2, entries=150, sequenceid=15, filesize=11.6 K 2023-04-19 20:55:40,903 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/C/b6ecd3d5a379435b990ecc11fae75469 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/b6ecd3d5a379435b990ecc11fae75469 2023-04-19 20:55:40,907 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/b6ecd3d5a379435b990ecc11fae75469, entries=150, sequenceid=15, filesize=11.6 K 2023-04-19 20:55:40,908 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=114.05 KB/116790 for 4393734ae917e967cf2b5b4b3b44ed0d in 603ms, sequenceid=15, compaction requested=false 2023-04-19 20:55:40,908 DEBUG [MemStoreFlusher.0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-04-19 20:55:40,909 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 4393734ae917e967cf2b5b4b3b44ed0d: 2023-04-19 20:55:40,947 DEBUG [Thread-3028] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:40,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:40,952 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:40,953 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681937740323, max timestamp=1681937740946Num uniques 859; 2023-04-19 20:55:40,953 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 20:55:40,953 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681937740323, max timestamp=1681937740946Num uniques 859; 2023-04-19 20:55:40,953 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 20:55:40,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:40,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:40,955 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:40,955 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681937740323, max timestamp=1681937740946Num uniques 859; 2023-04-19 20:55:40,955 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:40,955 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:40,955 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:40,955 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 20:55:40,956 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:40,956 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:40,958 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:40,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:40,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:40,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:40,958 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:40,959 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:40,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:40,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:40,959 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:40,959 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:40,959 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:40,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(9158): Flush requested on 4393734ae917e967cf2b5b4b3b44ed0d 2023-04-19 20:55:40,963 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 4393734ae917e967cf2b5b4b3b44ed0d 3/3 column families, dataSize=53.67 KB heapSize=145.92 KB 2023-04-19 20:55:40,964 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:55:40,964 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:40,964 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4393734ae917e967cf2b5b4b3b44ed0d, store=A 2023-04-19 20:55:40,964 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 20:55:40,964 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4393734ae917e967cf2b5b4b3b44ed0d, store=B 2023-04-19 20:55:40,964 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 20:55:40,964 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4393734ae917e967cf2b5b4b3b44ed0d, store=C 2023-04-19 20:55:40,964 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 20:55:40,964 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:40,964 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:40,964 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:40,969 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:40,969 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:40,969 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:40,969 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. started... 2023-04-19 20:55:40,970 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. as already flushing 2023-04-19 20:55:40,970 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:40,970 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:40,970 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:40,970 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:40,970 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:40,970 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:40,971 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:40,973 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:40,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:40,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:40,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:40,976 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:40,976 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:40,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:40,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:40,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:40,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:40,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:40,977 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:40,978 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@340fd921[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:40,978 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:40,978 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:40,979 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:40,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:40,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:40,980 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:40,980 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:40,980 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:40,980 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:40,980 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:40,981 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:40,981 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:40,981 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:40,981 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:40,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:40,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:40,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:40,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:40,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:40,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:40,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:40,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:40,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:40,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:40,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:40,986 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:40,986 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:40,986 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:40,986 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:40,986 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:40,986 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:40,987 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:40,987 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:40,987 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:40,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:40,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:40,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:40,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:40,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:40,989 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:40,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:40,989 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:40,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:40,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:40,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:40,991 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:40,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:40,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:40,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:40,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:40,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:40,993 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:40,993 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:40,993 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:40,993 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:40,993 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:40,993 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:40,994 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:40,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:40,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:40,994 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:40,993 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:40,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:40,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:40,994 DEBUG [Thread-3028] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:40,994 DEBUG [Thread-3028] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:40,994 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:40,994 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:40,994 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:40,995 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:55:41,002 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681937740969, max timestamp=1681937740999Num uniques 859; 2023-04-19 20:55:41,002 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681937740969, max timestamp=1681937740999Num uniques 859; 2023-04-19 20:55:41,002 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-19 20:55:41,002 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-19 20:55:41,003 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681937740969, max timestamp=1681937740999Num uniques 859; 2023-04-19 20:55:41,003 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-19 20:55:41,010 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=41 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/A/0e6df69316b3476995b3bbd854a55c3f 2023-04-19 20:55:41,031 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:41,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35486 deadline: 1681937801018, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:41,036 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:41,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35470 deadline: 1681937801025, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:41,036 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:41,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35490 deadline: 1681937801025, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:41,038 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=41 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/B/27dae2239ac84084b6d41bdb3cd6a0aa 2023-04-19 20:55:41,039 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:41,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35474 deadline: 1681937801026, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:41,039 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:41,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35492 deadline: 1681937801027, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:41,057 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=41 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/C/131e9ee789ff4ccc99ca836091dd6321 2023-04-19 20:55:41,062 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/A/0e6df69316b3476995b3bbd854a55c3f as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/0e6df69316b3476995b3bbd854a55c3f 2023-04-19 20:55:41,065 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/0e6df69316b3476995b3bbd854a55c3f, entries=250, sequenceid=41, filesize=16.2 K 2023-04-19 20:55:41,065 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/B/27dae2239ac84084b6d41bdb3cd6a0aa as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/27dae2239ac84084b6d41bdb3cd6a0aa 2023-04-19 20:55:41,069 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/27dae2239ac84084b6d41bdb3cd6a0aa, entries=150, sequenceid=41, filesize=11.6 K 2023-04-19 20:55:41,069 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/C/131e9ee789ff4ccc99ca836091dd6321 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/131e9ee789ff4ccc99ca836091dd6321 2023-04-19 20:55:41,075 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/131e9ee789ff4ccc99ca836091dd6321, entries=150, sequenceid=41, filesize=11.6 K 2023-04-19 20:55:41,076 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~165.80 KB/169776, currentSize=127.47 KB/130530 for 4393734ae917e967cf2b5b4b3b44ed0d in 113ms, sequenceid=41, compaction requested=false 2023-04-19 20:55:41,076 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 4393734ae917e967cf2b5b4b3b44ed0d: 2023-04-19 20:55:41,094 DEBUG [Thread-3028] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:41,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:41,096 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:41,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:41,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:41,104 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:41,104 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:41,104 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:41,104 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:41,104 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:41,104 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:41,107 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:41,107 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:41,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:41,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:41,107 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:41,107 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:41,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:41,108 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:41,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:41,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:41,108 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:41,112 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:55:41,112 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:41,112 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:41,112 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:41,112 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:41,113 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:41,113 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:41,113 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:41,113 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. started... 2023-04-19 20:55:41,113 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4393734ae917e967cf2b5b4b3b44ed0d 3/3 column families, dataSize=127.47 KB heapSize=355.69 KB 2023-04-19 20:55:41,114 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4393734ae917e967cf2b5b4b3b44ed0d, store=A 2023-04-19 20:55:41,114 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 20:55:41,114 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4393734ae917e967cf2b5b4b3b44ed0d, store=B 2023-04-19 20:55:41,114 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 20:55:41,114 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4393734ae917e967cf2b5b4b3b44ed0d, store=C 2023-04-19 20:55:41,114 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 20:55:41,138 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. as already flushing 2023-04-19 20:55:41,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(9158): Flush requested on 4393734ae917e967cf2b5b4b3b44ed0d 2023-04-19 20:55:41,154 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:41,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35490 deadline: 1681937801147, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:41,156 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:41,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35474 deadline: 1681937801147, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:41,157 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:41,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35492 deadline: 1681937801148, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:41,163 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:41,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35470 deadline: 1681937801154, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:41,163 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:41,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35486 deadline: 1681937801155, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:41,260 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:41,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35490 deadline: 1681937801256, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:41,260 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:41,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35492 deadline: 1681937801258, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:41,267 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:41,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35474 deadline: 1681937801258, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:41,271 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:41,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35486 deadline: 1681937801264, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:41,271 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:41,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35470 deadline: 1681937801265, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:41,469 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:41,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35490 deadline: 1681937801462, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:41,469 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:41,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35492 deadline: 1681937801462, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:41,475 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:41,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35474 deadline: 1681937801469, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:41,482 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:41,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35486 deadline: 1681937801473, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:41,482 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:41,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35470 deadline: 1681937801473, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:41,562 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=75 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/A/6508ddb2cff04e129cbb9f90ca8b10ef 2023-04-19 20:55:41,581 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=75 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/B/57970f5ab2d24c53b63b790d1a4dcd79 2023-04-19 20:55:41,606 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=75 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/C/5b670c0b836c4992894eb3acc1ab53d1 2023-04-19 20:55:41,612 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/A/6508ddb2cff04e129cbb9f90ca8b10ef as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/6508ddb2cff04e129cbb9f90ca8b10ef 2023-04-19 20:55:41,616 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/6508ddb2cff04e129cbb9f90ca8b10ef, entries=150, sequenceid=75, filesize=11.6 K 2023-04-19 20:55:41,616 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/B/57970f5ab2d24c53b63b790d1a4dcd79 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/57970f5ab2d24c53b63b790d1a4dcd79 2023-04-19 20:55:41,620 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/57970f5ab2d24c53b63b790d1a4dcd79, entries=150, sequenceid=75, filesize=11.6 K 2023-04-19 20:55:41,621 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/C/5b670c0b836c4992894eb3acc1ab53d1 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/5b670c0b836c4992894eb3acc1ab53d1 2023-04-19 20:55:41,637 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/5b670c0b836c4992894eb3acc1ab53d1, entries=150, sequenceid=75, filesize=11.6 K 2023-04-19 20:55:41,638 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~355.64 KB/364176, currentSize=53.67 KB/54960 for 4393734ae917e967cf2b5b4b3b44ed0d in 524ms, sequenceid=75, compaction requested=true 2023-04-19 20:55:41,639 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4393734ae917e967cf2b5b4b3b44ed0d: 2023-04-19 20:55:41,639 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:41,639 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:41,639 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:41,639 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:41,639 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:41,639 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:41,647 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:41,647 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:41,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:41,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:41,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:41,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:41,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:41,648 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:41,648 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:41,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:41,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:41,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:41,651 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:41,651 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2fe96518[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:41,651 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:41,651 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:41,652 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:41,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:41,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:41,654 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:41,654 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:41,654 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:41,654 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:41,654 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:41,661 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:41,661 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:41,661 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:41,661 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:41,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:41,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:41,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:41,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:41,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:41,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:41,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:41,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:41,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:41,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:41,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:41,664 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:41,664 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:41,664 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:41,664 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:41,664 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:41,664 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:41,667 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:41,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:41,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:41,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:41,667 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:41,667 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:41,667 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:41,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:41,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:41,668 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:41,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:41,668 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:41,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:41,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:41,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:41,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:41,670 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:41,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:41,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:41,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:41,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:41,672 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:41,672 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:41,672 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:41,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:41,672 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:41,672 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:41,672 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:41,673 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:41,673 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:41,673 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:41,673 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:41,673 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:41,673 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:41,673 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:41,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:41,673 DEBUG [Thread-3028] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:41,673 DEBUG [Thread-3028] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:41,672 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:41,673 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:55:41,773 DEBUG [Thread-3028] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:41,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:41,775 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:41,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:41,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:41,780 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:41,780 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:41,780 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:41,780 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:41,780 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:41,781 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:41,782 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:41,782 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:41,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:41,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:41,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:41,783 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:41,783 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:41,783 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:41,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:41,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:41,783 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:41,784 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:55:41,784 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:41,784 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:41,784 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:41,784 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:41,784 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:41,784 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:41,784 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:41,785 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. started... 2023-04-19 20:55:41,785 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4393734ae917e967cf2b5b4b3b44ed0d 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-19 20:55:41,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(9158): Flush requested on 4393734ae917e967cf2b5b4b3b44ed0d 2023-04-19 20:55:41,786 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. as already flushing 2023-04-19 20:55:41,786 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4393734ae917e967cf2b5b4b3b44ed0d, store=A 2023-04-19 20:55:41,787 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:55:41,787 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4393734ae917e967cf2b5b4b3b44ed0d, store=B 2023-04-19 20:55:41,787 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:55:41,787 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4393734ae917e967cf2b5b4b3b44ed0d, store=C 2023-04-19 20:55:41,787 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:55:41,818 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=88 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/A/fc4621db25f14d989e8f3460cfd0d42f 2023-04-19 20:55:41,822 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:41,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35486 deadline: 1681937801810, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:41,823 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:41,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35470 deadline: 1681937801811, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:41,823 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:41,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35490 deadline: 1681937801811, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:41,835 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:41,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35474 deadline: 1681937801822, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:41,836 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:41,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35492 deadline: 1681937801823, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:41,847 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=88 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/B/edda786758cb4b6c901a76f7c035e065 2023-04-19 20:55:41,873 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=88 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/C/f2d9e2014e9040f5a8c9922c97ba5f69 2023-04-19 20:55:41,879 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/A/fc4621db25f14d989e8f3460cfd0d42f as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/fc4621db25f14d989e8f3460cfd0d42f 2023-04-19 20:55:41,887 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/fc4621db25f14d989e8f3460cfd0d42f, entries=200, sequenceid=88, filesize=13.9 K 2023-04-19 20:55:41,888 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/B/edda786758cb4b6c901a76f7c035e065 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/edda786758cb4b6c901a76f7c035e065 2023-04-19 20:55:41,892 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/edda786758cb4b6c901a76f7c035e065, entries=150, sequenceid=88, filesize=11.6 K 2023-04-19 20:55:41,893 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/C/f2d9e2014e9040f5a8c9922c97ba5f69 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/f2d9e2014e9040f5a8c9922c97ba5f69 2023-04-19 20:55:41,896 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/f2d9e2014e9040f5a8c9922c97ba5f69, entries=150, sequenceid=88, filesize=11.6 K 2023-04-19 20:55:41,897 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 4393734ae917e967cf2b5b4b3b44ed0d in 112ms, sequenceid=88, compaction requested=true 2023-04-19 20:55:41,897 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4393734ae917e967cf2b5b4b3b44ed0d: 2023-04-19 20:55:41,897 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:41,897 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:41,897 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:41,897 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:41,897 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:41,897 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:41,901 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:41,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:41,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:41,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:41,901 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:41,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:41,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:41,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:41,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:41,902 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:41,902 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:41,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:41,903 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:41,903 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@595fcba2[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:41,903 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:41,903 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:41,905 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:41,906 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:41,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:41,906 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:41,906 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:41,906 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:41,906 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:41,906 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:41,908 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:41,908 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:41,908 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:41,908 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:41,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:41,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:41,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:41,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:41,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:41,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:41,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:41,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:41,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:41,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:41,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:41,912 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:41,912 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:41,912 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:41,912 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:41,912 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:41,912 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:41,916 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:41,916 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:41,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:41,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:41,916 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:41,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:41,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:41,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:41,917 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:41,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:41,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:41,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:41,918 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:41,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:41,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:41,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:41,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:41,920 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:41,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:41,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:41,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:41,924 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:41,924 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:41,924 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:41,924 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:41,924 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:41,924 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:41,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:41,924 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:41,924 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:41,924 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:41,924 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:41,924 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:41,924 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:41,924 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:41,925 DEBUG [Thread-3028] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:41,925 DEBUG [Thread-3028] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:41,925 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:41,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:41,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(9158): Flush requested on 4393734ae917e967cf2b5b4b3b44ed0d 2023-04-19 20:55:41,930 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 4393734ae917e967cf2b5b4b3b44ed0d 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-19 20:55:41,931 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4393734ae917e967cf2b5b4b3b44ed0d, store=A 2023-04-19 20:55:41,931 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:55:41,931 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4393734ae917e967cf2b5b4b3b44ed0d, store=B 2023-04-19 20:55:41,931 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:55:41,931 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4393734ae917e967cf2b5b4b3b44ed0d, store=C 2023-04-19 20:55:41,931 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:55:41,964 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:41,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35474 deadline: 1681937801951, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:41,964 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:41,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35490 deadline: 1681937801952, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:41,964 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:41,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35470 deadline: 1681937801955, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:41,968 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:41,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35486 deadline: 1681937801955, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:41,968 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:41,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35492 deadline: 1681937801964, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:41,981 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=108 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/A/115eea192f7b4b21a872c8f0c75a8813 2023-04-19 20:55:42,025 DEBUG [Thread-3028] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:42,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:42,027 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:42,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:42,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:42,030 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:42,030 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:42,030 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:42,030 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:42,031 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:42,031 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:42,033 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:42,033 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:42,033 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=108 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/B/54c4398026344169970b653f9f5362e3 2023-04-19 20:55:42,033 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:42,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:42,033 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:42,033 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:42,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:42,034 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:42,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:42,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:42,034 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:42,040 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:55:42,040 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:42,040 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:42,040 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:42,040 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:42,048 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:42,048 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:42,048 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:42,048 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. started... 2023-04-19 20:55:42,048 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. as already flushing 2023-04-19 20:55:42,048 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:42,048 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:42,048 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:42,048 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:42,048 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:42,048 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:42,061 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:42,061 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:42,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:42,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:42,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:42,062 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:42,062 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:42,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:42,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:42,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:42,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:42,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:42,065 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:42,066 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6762a123[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:42,066 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:42,066 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:42,069 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:42,069 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:42,069 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:42,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:42,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:42,071 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:42,071 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:42,071 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:42,071 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:42,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35474 deadline: 1681937802065, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:42,071 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:42,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35490 deadline: 1681937802065, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:42,071 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:42,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35470 deadline: 1681937802065, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:42,076 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:42,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:42,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:42,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:42,076 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:42,076 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:42,076 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:42,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:42,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:42,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:42,077 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:42,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35492 deadline: 1681937802069, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:42,078 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:42,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35486 deadline: 1681937802070, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:42,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:42,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:42,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:42,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:42,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:42,080 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:42,080 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:42,080 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:42,080 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:42,080 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:42,080 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:42,082 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:42,082 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:42,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:42,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:42,083 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:42,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:42,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:42,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:42,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:42,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:42,084 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:42,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:42,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:42,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:42,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:42,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:42,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:42,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:42,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:42,086 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:42,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:42,088 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:42,089 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:42,089 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:42,089 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:42,089 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:42,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:42,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:42,089 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:42,089 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:42,089 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:42,089 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:42,089 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:42,089 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:42,089 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:42,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:42,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:42,089 DEBUG [Thread-3028] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:42,089 DEBUG [Thread-3028] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:42,118 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=108 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/C/61468d6ab0ae4b31acd445825ed63fc8 2023-04-19 20:55:42,124 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/A/115eea192f7b4b21a872c8f0c75a8813 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/115eea192f7b4b21a872c8f0c75a8813 2023-04-19 20:55:42,137 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/115eea192f7b4b21a872c8f0c75a8813, entries=150, sequenceid=108, filesize=11.6 K 2023-04-19 20:55:42,138 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/B/54c4398026344169970b653f9f5362e3 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/54c4398026344169970b653f9f5362e3 2023-04-19 20:55:42,167 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/54c4398026344169970b653f9f5362e3, entries=150, sequenceid=108, filesize=11.6 K 2023-04-19 20:55:42,170 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/C/61468d6ab0ae4b31acd445825ed63fc8 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/61468d6ab0ae4b31acd445825ed63fc8 2023-04-19 20:55:42,174 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/61468d6ab0ae4b31acd445825ed63fc8, entries=150, sequenceid=108, filesize=11.6 K 2023-04-19 20:55:42,179 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 4393734ae917e967cf2b5b4b3b44ed0d in 249ms, sequenceid=108, compaction requested=true 2023-04-19 20:55:42,179 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 4393734ae917e967cf2b5b4b3b44ed0d: 2023-04-19 20:55:42,179 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 20:55:42,180 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 20:55:42,180 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-19 20:55:42,180 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 20:55:42,180 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-19 20:55:42,181 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59430 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-19 20:55:42,181 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1912): 4393734ae917e967cf2b5b4b3b44ed0d/B is initiating minor compaction (all files) 2023-04-19 20:55:42,181 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4393734ae917e967cf2b5b4b3b44ed0d/B in TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:42,181 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/c20e704eaa9047948b65bb0521962ae2, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/27dae2239ac84084b6d41bdb3cd6a0aa, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/57970f5ab2d24c53b63b790d1a4dcd79, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/edda786758cb4b6c901a76f7c035e065, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/54c4398026344169970b653f9f5362e3] into tmpdir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp, totalSize=58.0 K 2023-04-19 20:55:42,183 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 66450 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-19 20:55:42,183 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1912): 4393734ae917e967cf2b5b4b3b44ed0d/A is initiating minor compaction (all files) 2023-04-19 20:55:42,183 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 4393734ae917e967cf2b5b4b3b44ed0d/A in TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:42,183 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/593cad6360284a51a3288515b09116e0, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/0e6df69316b3476995b3bbd854a55c3f, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/6508ddb2cff04e129cbb9f90ca8b10ef, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/fc4621db25f14d989e8f3460cfd0d42f, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/115eea192f7b4b21a872c8f0c75a8813] into tmpdir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp, totalSize=64.9 K 2023-04-19 20:55:42,183 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting c20e704eaa9047948b65bb0521962ae2, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=15, earliestPutTs=1681937740298 2023-04-19 20:55:42,184 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 593cad6360284a51a3288515b09116e0, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=15, earliestPutTs=1681937740298 2023-04-19 20:55:42,184 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 0e6df69316b3476995b3bbd854a55c3f, keycount=250, bloomtype=ROW, size=16.2 K, encoding=NONE, compression=NONE, seqNum=41, earliestPutTs=1681937740323 2023-04-19 20:55:42,184 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 6508ddb2cff04e129cbb9f90ca8b10ef, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=75, earliestPutTs=1681937741017 2023-04-19 20:55:42,185 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting fc4621db25f14d989e8f3460cfd0d42f, keycount=200, bloomtype=ROW, size=13.9 K, encoding=NONE, compression=NONE, seqNum=88, earliestPutTs=1681937741147 2023-04-19 20:55:42,185 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 115eea192f7b4b21a872c8f0c75a8813, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=108, earliestPutTs=1681937741819 2023-04-19 20:55:42,185 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 27dae2239ac84084b6d41bdb3cd6a0aa, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=41, earliestPutTs=1681937740323 2023-04-19 20:55:42,190 DEBUG [Thread-3028] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:42,190 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 57970f5ab2d24c53b63b790d1a4dcd79, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=75, earliestPutTs=1681937741017 2023-04-19 20:55:42,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:42,191 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting edda786758cb4b6c901a76f7c035e065, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=88, earliestPutTs=1681937741147 2023-04-19 20:55:42,191 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 54c4398026344169970b653f9f5362e3, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=108, earliestPutTs=1681937741819 2023-04-19 20:55:42,195 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:42,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:42,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:42,212 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:42,212 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:42,212 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:42,212 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:42,213 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:42,213 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:42,214 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:42,214 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:42,214 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:42,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:42,214 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] throttle.PressureAwareThroughputController(145): 4393734ae917e967cf2b5b4b3b44ed0d#B#compaction#398 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 20:55:42,214 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:42,214 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:42,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:42,215 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:42,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:42,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:42,215 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:42,230 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] throttle.PressureAwareThroughputController(145): 4393734ae917e967cf2b5b4b3b44ed0d#A#compaction#399 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 20:55:42,255 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:55:42,255 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:42,255 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:42,255 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:42,255 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:42,255 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:42,255 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:42,255 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:42,256 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. started... 2023-04-19 20:55:42,256 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4393734ae917e967cf2b5b4b3b44ed0d 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-19 20:55:42,256 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4393734ae917e967cf2b5b4b3b44ed0d, store=A 2023-04-19 20:55:42,256 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:55:42,256 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4393734ae917e967cf2b5b4b3b44ed0d, store=B 2023-04-19 20:55:42,256 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:55:42,256 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4393734ae917e967cf2b5b4b3b44ed0d, store=C 2023-04-19 20:55:42,256 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:55:42,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(9158): Flush requested on 4393734ae917e967cf2b5b4b3b44ed0d 2023-04-19 20:55:42,282 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. as already flushing 2023-04-19 20:55:42,297 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/A/f19b87eaae6941969bb821a857ca9dc8 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/f19b87eaae6941969bb821a857ca9dc8 2023-04-19 20:55:42,297 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=120 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/A/b60ee764ac8e437091ed5a69bfaa3b13 2023-04-19 20:55:42,315 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 4393734ae917e967cf2b5b4b3b44ed0d/A of 4393734ae917e967cf2b5b4b3b44ed0d into f19b87eaae6941969bb821a857ca9dc8(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 20:55:42,315 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4393734ae917e967cf2b5b4b3b44ed0d: 2023-04-19 20:55:42,315 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d., storeName=4393734ae917e967cf2b5b4b3b44ed0d/A, priority=11, startTime=1681937742179; duration=0sec 2023-04-19 20:55:42,316 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 20:55:42,316 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-19 20:55:42,322 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59430 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-19 20:55:42,322 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1912): 4393734ae917e967cf2b5b4b3b44ed0d/C is initiating minor compaction (all files) 2023-04-19 20:55:42,322 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 4393734ae917e967cf2b5b4b3b44ed0d/C in TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:42,322 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/b6ecd3d5a379435b990ecc11fae75469, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/131e9ee789ff4ccc99ca836091dd6321, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/5b670c0b836c4992894eb3acc1ab53d1, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/f2d9e2014e9040f5a8c9922c97ba5f69, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/61468d6ab0ae4b31acd445825ed63fc8] into tmpdir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp, totalSize=58.0 K 2023-04-19 20:55:42,322 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting b6ecd3d5a379435b990ecc11fae75469, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=15, earliestPutTs=1681937740298 2023-04-19 20:55:42,323 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 131e9ee789ff4ccc99ca836091dd6321, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=41, earliestPutTs=1681937740323 2023-04-19 20:55:42,324 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 5b670c0b836c4992894eb3acc1ab53d1, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=75, earliestPutTs=1681937741017 2023-04-19 20:55:42,324 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting f2d9e2014e9040f5a8c9922c97ba5f69, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=88, earliestPutTs=1681937741147 2023-04-19 20:55:42,325 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 61468d6ab0ae4b31acd445825ed63fc8, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=108, earliestPutTs=1681937741819 2023-04-19 20:55:42,335 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] throttle.PressureAwareThroughputController(145): 4393734ae917e967cf2b5b4b3b44ed0d#C#compaction#402 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 20:55:42,345 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=120 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/B/7e5292be149e42b2a9fb18706abe24d7 2023-04-19 20:55:42,354 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/C/a458a37c08fa470ea7b666b009d5a409 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/a458a37c08fa470ea7b666b009d5a409 2023-04-19 20:55:42,359 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=120 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/C/9da508b6c036409b8668ece4023d04f4 2023-04-19 20:55:42,371 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/A/b60ee764ac8e437091ed5a69bfaa3b13 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/b60ee764ac8e437091ed5a69bfaa3b13 2023-04-19 20:55:42,377 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 4393734ae917e967cf2b5b4b3b44ed0d/C of 4393734ae917e967cf2b5b4b3b44ed0d into a458a37c08fa470ea7b666b009d5a409(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 20:55:42,377 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4393734ae917e967cf2b5b4b3b44ed0d: 2023-04-19 20:55:42,377 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d., storeName=4393734ae917e967cf2b5b4b3b44ed0d/C, priority=11, startTime=1681937742180; duration=0sec 2023-04-19 20:55:42,378 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 20:55:42,382 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/b60ee764ac8e437091ed5a69bfaa3b13, entries=150, sequenceid=120, filesize=11.6 K 2023-04-19 20:55:42,382 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/B/7e5292be149e42b2a9fb18706abe24d7 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/7e5292be149e42b2a9fb18706abe24d7 2023-04-19 20:55:42,387 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/7e5292be149e42b2a9fb18706abe24d7, entries=150, sequenceid=120, filesize=11.6 K 2023-04-19 20:55:42,387 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/C/9da508b6c036409b8668ece4023d04f4 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/9da508b6c036409b8668ece4023d04f4 2023-04-19 20:55:42,394 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/9da508b6c036409b8668ece4023d04f4, entries=150, sequenceid=120, filesize=11.6 K 2023-04-19 20:55:42,395 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=107.34 KB/109920 for 4393734ae917e967cf2b5b4b3b44ed0d in 139ms, sequenceid=120, compaction requested=false 2023-04-19 20:55:42,396 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4393734ae917e967cf2b5b4b3b44ed0d: 2023-04-19 20:55:42,396 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:42,396 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:42,396 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:42,396 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:42,396 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:42,396 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:42,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(9158): Flush requested on 4393734ae917e967cf2b5b4b3b44ed0d 2023-04-19 20:55:42,396 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 4393734ae917e967cf2b5b4b3b44ed0d 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-19 20:55:42,398 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681937742312, max timestamp=1681937742395Num uniques 859; 2023-04-19 20:55:42,398 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-19 20:55:42,400 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681937742312, max timestamp=1681937742395Num uniques 859; 2023-04-19 20:55:42,400 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 20:55:42,400 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:42,400 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:42,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:42,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:42,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:42,401 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:42,401 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:42,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:42,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:42,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:42,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:42,402 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681937742312, max timestamp=1681937742395Num uniques 859; 2023-04-19 20:55:42,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:42,402 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 20:55:42,402 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:42,402 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4393734ae917e967cf2b5b4b3b44ed0d, store=A 2023-04-19 20:55:42,402 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:42,402 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 20:55:42,402 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2deeaa3f[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:42,402 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4393734ae917e967cf2b5b4b3b44ed0d, store=B 2023-04-19 20:55:42,402 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 20:55:42,402 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:42,403 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4393734ae917e967cf2b5b4b3b44ed0d, store=C 2023-04-19 20:55:42,403 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 20:55:42,404 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:42,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:42,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:42,404 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:42,404 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:42,404 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:42,405 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:42,405 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:42,406 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:42,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:42,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:42,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:42,406 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:42,406 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:42,406 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:42,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:42,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:42,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:42,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:42,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:42,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:42,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:42,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:42,409 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:42,409 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:42,409 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:42,409 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:42,409 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:42,409 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:42,410 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:42,410 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:42,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:42,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:42,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:42,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:42,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:42,411 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:42,412 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:42,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:42,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:42,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:42,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:42,415 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:42,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:42,418 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:42,418 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:42,418 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:42,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:42,418 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:42,418 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:42,418 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:42,418 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:42,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:42,418 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:42,418 DEBUG [Thread-3028] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:42,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:42,418 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:42,418 DEBUG [Thread-3028] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:42,419 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:42,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:42,419 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:42,419 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:42,419 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:42,419 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:42,419 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:55:42,424 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=143 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/A/e87262c540ef4372b3ec73f498a60b28 2023-04-19 20:55:42,485 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681937742450, max timestamp=1681937742475Num uniques 859; 2023-04-19 20:55:42,485 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 20:55:42,487 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681937742450, max timestamp=1681937742475Num uniques 859; 2023-04-19 20:55:42,487 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 20:55:42,488 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681937742450, max timestamp=1681937742475Num uniques 859; 2023-04-19 20:55:42,490 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 20:55:42,519 DEBUG [Thread-3028] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:42,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:42,521 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:42,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:42,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:42,523 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:42,523 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:42,523 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:42,523 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:42,523 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:42,523 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:42,525 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:42,525 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:42,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:42,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:42,525 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:42,525 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:42,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:42,526 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:42,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:42,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:42,526 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:42,528 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:55:42,528 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:42,528 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:42,528 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:42,528 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:42,532 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:42,532 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:42,532 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:42,532 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. started... 2023-04-19 20:55:42,532 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. as already flushing 2023-04-19 20:55:42,532 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:42,532 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:42,532 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:42,532 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:42,533 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:42,533 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:42,534 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:42,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:42,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:42,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:42,535 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:42,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:42,535 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:42,535 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:42,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:42,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:42,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:42,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:42,537 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:42,537 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@49cbdb3f[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:42,537 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:42,537 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:42,538 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:42,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:42,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:42,539 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:42,539 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:42,539 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:42,539 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:42,539 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:42,541 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:42,541 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:42,541 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:42,541 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:42,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:42,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:42,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:42,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:42,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:42,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:42,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:42,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:42,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:42,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:42,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:42,543 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:42,543 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:42,543 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:42,543 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:42,543 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:42,543 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:42,547 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:42,547 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:42,547 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:42,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:42,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:42,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:42,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:42,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:42,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:42,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:42,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:42,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:42,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:42,549 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:42,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:42,551 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1681937742511, max timestamp=1681937742548Num uniques 1018; 2023-04-19 20:55:42,551 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=12268, suffix heap size=133840,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=268 2023-04-19 20:55:42,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:42,553 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1681937742511, max timestamp=1681937742548Num uniques 1018; 2023-04-19 20:55:42,553 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=12268, suffix heap size=133840,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=268 2023-04-19 20:55:42,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:42,553 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1681937742511, max timestamp=1681937742548Num uniques 1018; 2023-04-19 20:55:42,553 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=12268, suffix heap size=133840,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=268 2023-04-19 20:55:42,554 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:42,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:42,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:42,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:42,560 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:42,560 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:42,560 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:42,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:42,560 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:42,560 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:42,560 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:42,560 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:42,560 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:42,560 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:42,560 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:42,560 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:42,560 DEBUG [Thread-3028] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:42,560 DEBUG [Thread-3028] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:42,560 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:42,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:42,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:42,561 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:42,621 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=327, cellSize=14972, totalHeapSize=34248, min timestamp=1681937742605, max timestamp=1681937742618Num uniques 1127; 2023-04-19 20:55:42,621 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=14972, suffix heap size=145176,new segment heap size=34248 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=327 2023-04-19 20:55:42,622 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=327, cellSize=14972, totalHeapSize=34248, min timestamp=1681937742605, max timestamp=1681937742618Num uniques 1127; 2023-04-19 20:55:42,622 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=327, cellSize=14972, totalHeapSize=34248, min timestamp=1681937742605, max timestamp=1681937742618Num uniques 1127; 2023-04-19 20:55:42,622 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=14972, suffix heap size=145176,new segment heap size=34248 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=327 2023-04-19 20:55:42,622 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=14972, suffix heap size=145176,new segment heap size=34248 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=327 2023-04-19 20:55:42,661 DEBUG [Thread-3028] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:42,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:42,666 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:42,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35474 deadline: 1681937802655, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:42,667 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=336, cellSize=15386, totalHeapSize=35184, min timestamp=1681937742642, max timestamp=1681937742664Num uniques 1186; 2023-04-19 20:55:42,667 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=54316, new segment data size=15386, suffix heap size=151312,new segment heap size=35184 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1186, new segment cells count=336 2023-04-19 20:55:42,668 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=336, cellSize=15386, totalHeapSize=35184, min timestamp=1681937742642, max timestamp=1681937742664Num uniques 1186; 2023-04-19 20:55:42,668 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=54316, new segment data size=15386, suffix heap size=151312,new segment heap size=35184 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1186, new segment cells count=336 2023-04-19 20:55:42,669 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=336, cellSize=15386, totalHeapSize=35184, min timestamp=1681937742642, max timestamp=1681937742664Num uniques 1186; 2023-04-19 20:55:42,670 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=54316, new segment data size=15386, suffix heap size=151312,new segment heap size=35184 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1186, new segment cells count=336 2023-04-19 20:55:42,670 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/B/7f0298a614364121a886b61a788cffc5 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/7f0298a614364121a886b61a788cffc5 2023-04-19 20:55:42,670 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:42,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:42,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:42,673 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:42,673 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:42,673 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:42,673 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:42,674 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:42,674 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:42,676 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:42,676 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:42,676 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:42,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:42,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:42,677 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:42,677 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:42,677 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:42,677 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 4393734ae917e967cf2b5b4b3b44ed0d/B of 4393734ae917e967cf2b5b4b3b44ed0d into 7f0298a614364121a886b61a788cffc5(size=11.8 K), total size for store is 23.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 20:55:42,677 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4393734ae917e967cf2b5b4b3b44ed0d: 2023-04-19 20:55:42,678 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d., storeName=4393734ae917e967cf2b5b4b3b44ed0d/B, priority=11, startTime=1681937742180; duration=0sec 2023-04-19 20:55:42,678 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 20:55:42,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:42,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:42,678 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:42,680 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:55:42,680 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:42,680 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:42,680 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:42,680 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:42,684 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:42,684 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:42,684 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:42,684 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. started... 2023-04-19 20:55:42,684 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. as already flushing 2023-04-19 20:55:42,684 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:42,684 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:42,684 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:42,684 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:42,684 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:42,684 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:42,686 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:42,686 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:42,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:42,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:42,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:42,686 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:42,686 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:42,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:42,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:42,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:42,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:42,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:42,689 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:42,689 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6eee0b7f[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:42,689 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:42,690 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:42,692 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:42,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:42,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:42,693 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:42,693 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:42,693 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:42,693 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:42,693 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:42,694 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:42,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:42,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:42,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:42,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:42,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:42,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:42,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:42,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:42,696 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:42,696 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:42,696 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:42,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:42,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:42,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:42,697 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:42,697 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:42,697 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:42,697 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:42,697 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:42,698 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:42,702 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:42,702 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:42,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:42,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:42,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:42,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:42,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:42,702 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:42,702 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:42,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:42,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:42,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:42,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:42,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:42,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:42,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:42,704 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:42,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:42,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:42,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:42,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:42,706 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:42,706 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:42,706 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:42,706 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:42,706 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:42,706 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:42,706 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:42,706 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:42,706 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:42,707 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:42,706 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:42,707 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:42,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:42,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:42,706 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:55:42,707 DEBUG [Thread-3028] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:42,707 DEBUG [Thread-3028] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:42,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:42,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:42,720 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:42,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35470 deadline: 1681937802713, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:42,720 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=295, cellSize=13510, totalHeapSize=30920, min timestamp=1681937742688, max timestamp=1681937742718Num uniques 1195; 2023-04-19 20:55:42,720 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=54730, new segment data size=13510, suffix heap size=152248,new segment heap size=30920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1195, new segment cells count=295 2023-04-19 20:55:42,721 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=295, cellSize=13510, totalHeapSize=30920, min timestamp=1681937742688, max timestamp=1681937742718Num uniques 1195; 2023-04-19 20:55:42,721 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=54730, new segment data size=13510, suffix heap size=152248,new segment heap size=30920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1195, new segment cells count=295 2023-04-19 20:55:42,721 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=295, cellSize=13510, totalHeapSize=30920, min timestamp=1681937742688, max timestamp=1681937742718Num uniques 1195; 2023-04-19 20:55:42,721 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=54730, new segment data size=13510, suffix heap size=152248,new segment heap size=30920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1195, new segment cells count=295 2023-04-19 20:55:42,790 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=254, cellSize=11630, totalHeapSize=26656, min timestamp=1681937742761, max timestamp=1681937742787Num uniques 1154; 2023-04-19 20:55:42,790 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=52850, new segment data size=11630, suffix heap size=147984,new segment heap size=26656 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1154, new segment cells count=254 2023-04-19 20:55:42,790 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=254, cellSize=11630, totalHeapSize=26656, min timestamp=1681937742761, max timestamp=1681937742787Num uniques 1154; 2023-04-19 20:55:42,790 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=52850, new segment data size=11630, suffix heap size=147984,new segment heap size=26656 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1154, new segment cells count=254 2023-04-19 20:55:42,790 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=254, cellSize=11624, totalHeapSize=26656, min timestamp=1681937742761, max timestamp=1681937742787Num uniques 1154; 2023-04-19 20:55:42,790 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=52844, new segment data size=11624, suffix heap size=147984,new segment heap size=26656 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1154, new segment cells count=254 2023-04-19 20:55:42,807 DEBUG [Thread-3028] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:42,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:42,810 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:42,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:42,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:42,811 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:42,811 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:42,811 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:42,811 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:42,812 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:42,812 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:42,813 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:42,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:42,814 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:42,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:42,814 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:42,814 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:42,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:42,814 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:42,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:42,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:42,815 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:42,815 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:55:42,815 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:42,815 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:42,815 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:42,815 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:42,815 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:42,815 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:42,815 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:42,816 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. started... 2023-04-19 20:55:42,816 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. as already flushing 2023-04-19 20:55:42,816 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:42,816 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:42,816 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:42,816 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:42,816 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:42,816 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:42,817 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:42,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:42,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:42,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:42,817 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:42,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:42,818 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:42,818 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:42,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:42,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:42,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:42,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:42,819 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:42,819 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1c77d438[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:42,819 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:42,819 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:42,821 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:42,821 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:42,824 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:42,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:42,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:42,824 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:42,824 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:42,824 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:42,826 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:42,826 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:42,826 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:42,826 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:42,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:42,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:42,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:42,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:42,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:42,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:42,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:42,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:42,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:42,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:42,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:42,829 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:42,829 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:42,830 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:42,830 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:42,830 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:42,830 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:42,831 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:42,831 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:42,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:42,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:42,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:42,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:42,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:42,831 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:42,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:42,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:42,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:42,833 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:42,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:42,835 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1681937742814, max timestamp=1681937742831Num uniques 1113; 2023-04-19 20:55:42,835 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=50968, new segment data size=9748, suffix heap size=143720,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1113, new segment cells count=213 2023-04-19 20:55:42,835 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1681937742814, max timestamp=1681937742831Num uniques 1113; 2023-04-19 20:55:42,835 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=50968, new segment data size=9748, suffix heap size=143720,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1113, new segment cells count=213 2023-04-19 20:55:42,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:42,835 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1681937742814, max timestamp=1681937742831Num uniques 1113; 2023-04-19 20:55:42,835 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=50968, new segment data size=9748, suffix heap size=143720,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1113, new segment cells count=213 2023-04-19 20:55:42,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:42,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:42,838 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:42,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:42,840 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:42,840 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:42,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:42,840 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:42,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:42,840 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:42,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:42,840 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:42,840 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:42,840 DEBUG [Thread-3028] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:42,841 DEBUG [Thread-3028] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:42,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:42,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:42,840 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:42,841 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:42,841 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:42,841 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:42,841 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:42,841 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:42,843 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:55:42,874 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=372, cellSize=17032, totalHeapSize=38928, min timestamp=1681937742851, max timestamp=1681937742871Num uniques 1072; 2023-04-19 20:55:42,874 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=17032, suffix heap size=139456,new segment heap size=38928 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=372 2023-04-19 20:55:42,874 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=372, cellSize=17032, totalHeapSize=38928, min timestamp=1681937742851, max timestamp=1681937742871Num uniques 1072; 2023-04-19 20:55:42,874 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=17032, suffix heap size=139456,new segment heap size=38928 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=372 2023-04-19 20:55:42,874 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=372, cellSize=17032, totalHeapSize=38928, min timestamp=1681937742851, max timestamp=1681937742871Num uniques 1072; 2023-04-19 20:55:42,875 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=17032, suffix heap size=139456,new segment heap size=38928 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=372 2023-04-19 20:55:42,891 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=143 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/B/e763ddb01c92427db22e8f6a441d3bcf 2023-04-19 20:55:42,903 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:42,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35492 deadline: 1681937802894, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:42,903 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:42,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35470 deadline: 1681937802896, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:42,906 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=143 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/C/86989a3cc20441d896ad38e61fb52150 2023-04-19 20:55:42,909 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:42,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35490 deadline: 1681937802897, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:42,909 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:42,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35474 deadline: 1681937802898, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:42,909 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:42,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35486 deadline: 1681937802901, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:42,911 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/A/e87262c540ef4372b3ec73f498a60b28 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/e87262c540ef4372b3ec73f498a60b28 2023-04-19 20:55:42,916 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/e87262c540ef4372b3ec73f498a60b28, entries=200, sequenceid=143, filesize=14.1 K 2023-04-19 20:55:42,916 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/B/e763ddb01c92427db22e8f6a441d3bcf as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/e763ddb01c92427db22e8f6a441d3bcf 2023-04-19 20:55:42,923 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/e763ddb01c92427db22e8f6a441d3bcf, entries=150, sequenceid=143, filesize=11.8 K 2023-04-19 20:55:42,925 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/C/86989a3cc20441d896ad38e61fb52150 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/86989a3cc20441d896ad38e61fb52150 2023-04-19 20:55:42,929 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/86989a3cc20441d896ad38e61fb52150, entries=150, sequenceid=143, filesize=11.8 K 2023-04-19 20:55:42,930 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~31.31 KB/32060, heapSize ~76.34 KB/78176, currentSize=161.02 KB/164880 for 4393734ae917e967cf2b5b4b3b44ed0d in 534ms, sequenceid=143, compaction requested=true 2023-04-19 20:55:42,930 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 4393734ae917e967cf2b5b4b3b44ed0d: 2023-04-19 20:55:42,931 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 20:55:42,931 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-19 20:55:42,931 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 20:55:42,931 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-19 20:55:42,931 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 20:55:42,931 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 38370 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-19 20:55:42,932 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1912): 4393734ae917e967cf2b5b4b3b44ed0d/A is initiating minor compaction (all files) 2023-04-19 20:55:42,932 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35980 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-19 20:55:42,932 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 4393734ae917e967cf2b5b4b3b44ed0d/A in TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:42,932 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1912): 4393734ae917e967cf2b5b4b3b44ed0d/B is initiating minor compaction (all files) 2023-04-19 20:55:42,932 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/f19b87eaae6941969bb821a857ca9dc8, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/b60ee764ac8e437091ed5a69bfaa3b13, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/e87262c540ef4372b3ec73f498a60b28] into tmpdir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp, totalSize=37.5 K 2023-04-19 20:55:42,932 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4393734ae917e967cf2b5b4b3b44ed0d/B in TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:42,932 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/7f0298a614364121a886b61a788cffc5, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/7e5292be149e42b2a9fb18706abe24d7, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/e763ddb01c92427db22e8f6a441d3bcf] into tmpdir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp, totalSize=35.1 K 2023-04-19 20:55:42,932 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting f19b87eaae6941969bb821a857ca9dc8, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=108, earliestPutTs=1681937741819 2023-04-19 20:55:42,932 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 7f0298a614364121a886b61a788cffc5, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=108, earliestPutTs=1681937741819 2023-04-19 20:55:42,934 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting b60ee764ac8e437091ed5a69bfaa3b13, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=120, earliestPutTs=1681937741951 2023-04-19 20:55:42,934 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 7e5292be149e42b2a9fb18706abe24d7, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=120, earliestPutTs=1681937741951 2023-04-19 20:55:42,936 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting e87262c540ef4372b3ec73f498a60b28, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=143, earliestPutTs=1681937742312 2023-04-19 20:55:42,936 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting e763ddb01c92427db22e8f6a441d3bcf, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=143, earliestPutTs=1681937742312 2023-04-19 20:55:42,941 DEBUG [Thread-3028] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:42,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:42,943 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:42,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:42,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:42,948 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:42,948 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:42,948 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:42,948 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:42,948 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:42,949 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:42,951 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:42,951 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:42,951 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:42,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:42,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:42,951 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:42,952 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:42,952 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:42,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:42,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:42,952 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:42,956 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] throttle.PressureAwareThroughputController(145): 4393734ae917e967cf2b5b4b3b44ed0d#B#compaction#407 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 20:55:42,957 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:55:42,957 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:42,957 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:42,957 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:42,957 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:42,958 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] throttle.PressureAwareThroughputController(145): 4393734ae917e967cf2b5b4b3b44ed0d#A#compaction#408 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 20:55:42,960 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:42,960 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:42,960 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:42,961 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. started... 2023-04-19 20:55:42,962 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4393734ae917e967cf2b5b4b3b44ed0d 3/3 column families, dataSize=161.02 KB heapSize=444.70 KB 2023-04-19 20:55:42,962 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4393734ae917e967cf2b5b4b3b44ed0d, store=A 2023-04-19 20:55:42,962 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 20:55:42,962 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4393734ae917e967cf2b5b4b3b44ed0d, store=B 2023-04-19 20:55:42,962 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 20:55:42,962 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4393734ae917e967cf2b5b4b3b44ed0d, store=C 2023-04-19 20:55:42,962 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 20:55:42,985 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/B/a86e2147024c4edfad3b8b32a904ebbe as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/a86e2147024c4edfad3b8b32a904ebbe 2023-04-19 20:55:42,989 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 4393734ae917e967cf2b5b4b3b44ed0d/B of 4393734ae917e967cf2b5b4b3b44ed0d into a86e2147024c4edfad3b8b32a904ebbe(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 20:55:42,989 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4393734ae917e967cf2b5b4b3b44ed0d: 2023-04-19 20:55:42,989 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d., storeName=4393734ae917e967cf2b5b4b3b44ed0d/B, priority=13, startTime=1681937742931; duration=0sec 2023-04-19 20:55:42,990 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 20:55:42,990 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-19 20:55:42,991 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=301 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/A/9ea01a31f7cb415ebbef0aa052832786 2023-04-19 20:55:42,991 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35980 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-19 20:55:42,991 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1912): 4393734ae917e967cf2b5b4b3b44ed0d/C is initiating minor compaction (all files) 2023-04-19 20:55:42,991 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4393734ae917e967cf2b5b4b3b44ed0d/C in TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:42,991 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/a458a37c08fa470ea7b666b009d5a409, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/9da508b6c036409b8668ece4023d04f4, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/86989a3cc20441d896ad38e61fb52150] into tmpdir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp, totalSize=35.1 K 2023-04-19 20:55:42,993 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting a458a37c08fa470ea7b666b009d5a409, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=108, earliestPutTs=1681937741819 2023-04-19 20:55:42,993 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 9da508b6c036409b8668ece4023d04f4, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=120, earliestPutTs=1681937741951 2023-04-19 20:55:42,994 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 86989a3cc20441d896ad38e61fb52150, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=143, earliestPutTs=1681937742312 2023-04-19 20:55:43,006 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] throttle.PressureAwareThroughputController(145): 4393734ae917e967cf2b5b4b3b44ed0d#C#compaction#411 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 20:55:43,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(9158): Flush requested on 4393734ae917e967cf2b5b4b3b44ed0d 2023-04-19 20:55:43,012 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. as already flushing 2023-04-19 20:55:43,016 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:43,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35486 deadline: 1681937803010, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:43,019 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:43,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35492 deadline: 1681937803013, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:43,019 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:43,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35470 deadline: 1681937803014, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:43,023 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:43,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35474 deadline: 1681937803016, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:43,023 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:43,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35490 deadline: 1681937803017, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:43,032 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=301 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/B/01c9a67084494cec85710db4418b8cc3 2023-04-19 20:55:43,046 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=301 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/C/b9e3de9a3ac541f3be7ed560f00da299 2023-04-19 20:55:43,050 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/A/9ea01a31f7cb415ebbef0aa052832786 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/9ea01a31f7cb415ebbef0aa052832786 2023-04-19 20:55:43,055 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/9ea01a31f7cb415ebbef0aa052832786, entries=150, sequenceid=301, filesize=11.9 K 2023-04-19 20:55:43,056 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/B/01c9a67084494cec85710db4418b8cc3 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/01c9a67084494cec85710db4418b8cc3 2023-04-19 20:55:43,060 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/01c9a67084494cec85710db4418b8cc3, entries=150, sequenceid=301, filesize=11.9 K 2023-04-19 20:55:43,061 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/C/b9e3de9a3ac541f3be7ed560f00da299 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/b9e3de9a3ac541f3be7ed560f00da299 2023-04-19 20:55:43,063 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/b9e3de9a3ac541f3be7ed560f00da299, entries=150, sequenceid=301, filesize=11.9 K 2023-04-19 20:55:43,064 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~161.02 KB/164880, heapSize ~444.66 KB/455328, currentSize=26.84 KB/27480 for 4393734ae917e967cf2b5b4b3b44ed0d in 102ms, sequenceid=301, compaction requested=false 2023-04-19 20:55:43,064 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4393734ae917e967cf2b5b4b3b44ed0d: 2023-04-19 20:55:43,064 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:43,064 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:43,064 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:43,064 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:43,064 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:43,064 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:43,066 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:43,066 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:43,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:43,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:43,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:43,067 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:43,067 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:43,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:43,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:43,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:43,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:43,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:43,069 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:43,069 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@44102adb[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:43,069 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:43,069 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:43,071 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:43,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:43,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:43,071 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:43,071 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:43,071 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:43,071 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:43,071 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:43,072 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:43,073 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:43,073 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:43,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:43,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:43,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:43,073 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:43,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:43,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:43,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:43,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:43,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:43,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:43,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:43,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:43,075 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:43,075 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:43,075 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:43,075 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:43,075 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:43,075 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:43,080 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:43,080 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:43,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:43,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:43,080 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:43,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:43,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:43,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:43,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:43,081 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:43,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:43,081 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:43,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:43,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:43,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:43,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:43,082 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:43,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:43,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:43,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:43,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:43,086 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:43,086 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:43,086 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:43,086 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:43,086 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:43,086 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:43,087 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:43,087 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:43,087 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:43,087 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:43,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:43,087 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:43,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:43,087 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:55:43,087 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:43,088 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:43,088 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:43,089 DEBUG [Thread-3028] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:43,089 DEBUG [Thread-3028] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:43,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(9158): Flush requested on 4393734ae917e967cf2b5b4b3b44ed0d 2023-04-19 20:55:43,131 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 4393734ae917e967cf2b5b4b3b44ed0d 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-19 20:55:43,132 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4393734ae917e967cf2b5b4b3b44ed0d, store=A 2023-04-19 20:55:43,132 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:55:43,132 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4393734ae917e967cf2b5b4b3b44ed0d, store=B 2023-04-19 20:55:43,132 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:55:43,132 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4393734ae917e967cf2b5b4b3b44ed0d, store=C 2023-04-19 20:55:43,132 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:55:43,175 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:43,175 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:43,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35470 deadline: 1681937803166, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:43,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35492 deadline: 1681937803167, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:43,175 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:43,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35490 deadline: 1681937803174, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:43,181 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:43,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35474 deadline: 1681937803175, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:43,189 DEBUG [Thread-3028] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:43,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:43,191 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:43,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:43,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:43,192 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:43,192 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:43,192 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:43,192 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:43,192 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:43,192 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:43,196 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:43,197 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:43,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:43,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:43,197 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:43,197 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:43,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:43,197 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:43,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:43,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:43,198 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:43,198 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:55:43,198 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:43,198 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:43,198 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:43,198 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:43,198 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:43,198 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:43,198 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:43,198 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. started... 2023-04-19 20:55:43,198 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. as already flushing 2023-04-19 20:55:43,198 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:43,198 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:43,198 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:43,199 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:43,199 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:43,199 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:43,201 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:43,201 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:43,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:43,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:43,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:43,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:43,201 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:43,201 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:43,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:43,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:43,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:43,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:43,202 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:43,202 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@53151345[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:43,202 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:43,203 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:43,204 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:43,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:43,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:43,204 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:43,204 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:43,204 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:43,204 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:43,204 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:43,206 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:43,206 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:43,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:43,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:43,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:43,206 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:43,206 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:43,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:43,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:43,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:43,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:43,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:43,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:43,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:43,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:43,209 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:43,209 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:43,209 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:43,209 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:43,209 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:43,209 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:43,211 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:43,211 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:43,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:43,211 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:43,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:43,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:43,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:43,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:43,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:43,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:43,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:43,212 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:43,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:43,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:43,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:43,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:43,213 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:43,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:43,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:43,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:43,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:43,216 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:43,217 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:43,217 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:43,217 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:43,217 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:43,217 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:43,217 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:43,217 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:43,217 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:43,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:43,217 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:55:43,217 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:43,217 DEBUG [Thread-3028] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:43,217 DEBUG [Thread-3028] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:43,217 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:43,217 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:43,217 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:43,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:43,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:43,226 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:43,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35486 deadline: 1681937803219, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:43,280 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:43,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35470 deadline: 1681937803276, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:43,281 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:43,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35492 deadline: 1681937803276, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:43,281 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:43,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35490 deadline: 1681937803276, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:43,288 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:43,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35474 deadline: 1681937803282, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:43,317 DEBUG [Thread-3028] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:43,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:43,319 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:43,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:43,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:43,320 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:43,320 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:43,320 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:43,320 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:43,321 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:43,321 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:43,322 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:43,322 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:43,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:43,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:43,322 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:43,322 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:43,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:43,322 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:43,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:43,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:43,323 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:43,323 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:55:43,323 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:43,323 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:43,323 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:43,323 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:43,323 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:43,323 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:43,323 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:43,323 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. started... 2023-04-19 20:55:43,323 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. as already flushing 2023-04-19 20:55:43,324 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:43,324 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:43,324 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:43,324 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:43,324 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:43,324 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:43,325 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:43,325 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:43,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:43,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:43,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:43,325 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:43,325 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:43,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:43,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:43,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:43,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:43,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:43,326 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:43,327 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4dbe5fab[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:43,327 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:43,327 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:43,328 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:43,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:43,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:43,328 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:43,328 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:43,328 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:43,328 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:43,328 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:43,330 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:43,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:43,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:43,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:43,330 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:43,330 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:43,330 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:43,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:43,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:43,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:43,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:43,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:43,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:43,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:43,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:43,333 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:43,333 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:43,333 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:43,333 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:43,333 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:43,333 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:43,335 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:43,335 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:43,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:43,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:43,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:43,335 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:43,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:43,335 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:43,335 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:43,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:43,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:43,335 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:43,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:43,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:43,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:43,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:43,336 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:43,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:43,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:43,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:43,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:43,338 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:43,338 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:43,338 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:43,338 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:43,338 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:43,338 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:43,338 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:43,339 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:43,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:43,339 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:55:43,339 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:43,338 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:43,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:43,339 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:43,339 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:43,339 DEBUG [Thread-3028] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:43,339 DEBUG [Thread-3028] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:43,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:43,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:43,382 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/A/bdea0565ed934118b46236767cc2487b as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/bdea0565ed934118b46236767cc2487b 2023-04-19 20:55:43,386 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 4393734ae917e967cf2b5b4b3b44ed0d/A of 4393734ae917e967cf2b5b4b3b44ed0d into bdea0565ed934118b46236767cc2487b(size=12.0 K), total size for store is 23.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 20:55:43,386 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4393734ae917e967cf2b5b4b3b44ed0d: 2023-04-19 20:55:43,386 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d., storeName=4393734ae917e967cf2b5b4b3b44ed0d/A, priority=13, startTime=1681937742931; duration=0sec 2023-04-19 20:55:43,386 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 20:55:43,429 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/C/b616a1072bfb42f4928243d32fbd9d65 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/b616a1072bfb42f4928243d32fbd9d65 2023-04-19 20:55:43,433 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 4393734ae917e967cf2b5b4b3b44ed0d/C of 4393734ae917e967cf2b5b4b3b44ed0d into b616a1072bfb42f4928243d32fbd9d65(size=12.0 K), total size for store is 23.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 20:55:43,433 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4393734ae917e967cf2b5b4b3b44ed0d: 2023-04-19 20:55:43,433 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d., storeName=4393734ae917e967cf2b5b4b3b44ed0d/C, priority=13, startTime=1681937742931; duration=0sec 2023-04-19 20:55:43,433 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 20:55:43,439 DEBUG [Thread-3028] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:43,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:43,441 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:43,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:43,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:43,442 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:43,442 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:43,442 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:43,442 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:43,443 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:43,443 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:43,444 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:43,444 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:43,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:43,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:43,444 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:43,444 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:43,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:43,445 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:43,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:43,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:43,445 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:43,445 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:55:43,445 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:43,445 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:43,445 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:43,445 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:43,446 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:43,446 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:43,446 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:43,446 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. started... 2023-04-19 20:55:43,446 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. as already flushing 2023-04-19 20:55:43,446 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:43,446 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:43,446 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:43,446 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:43,446 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:43,446 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:43,448 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:43,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:43,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:43,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:43,448 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:43,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:43,448 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:43,449 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:43,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:43,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:43,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:43,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:43,450 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:43,450 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6f4a9430[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:43,450 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:43,450 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:43,451 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:43,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:43,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:43,451 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:43,451 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:43,451 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:43,451 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:43,451 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:43,452 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:43,452 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:43,452 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:43,453 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:43,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:43,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:43,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:43,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:43,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:43,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:43,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:43,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:43,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:43,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:43,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:43,455 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:43,455 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:43,455 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:43,455 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:43,455 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:43,455 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:43,459 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:43,459 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:43,459 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:43,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:43,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:43,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:43,459 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:43,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:43,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:43,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:43,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:43,460 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:43,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:43,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:43,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:43,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:43,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:43,461 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:43,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:43,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:43,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:43,464 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:43,464 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:43,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:43,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:43,464 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:43,464 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:43,464 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:43,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:43,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:43,464 DEBUG [Thread-3028] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:43,465 DEBUG [Thread-3028] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:43,464 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:43,465 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:43,465 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:43,465 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:43,465 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:43,465 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:43,465 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:43,486 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:43,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35492 deadline: 1681937803482, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:43,486 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:43,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35470 deadline: 1681937803482, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:43,486 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:43,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35490 deadline: 1681937803483, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:43,495 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:43,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35474 deadline: 1681937803489, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:43,533 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:43,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35486 deadline: 1681937803529, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:43,550 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=314 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/A/9a91bef8a8544db6a38a37d8b1eb6d58 2023-04-19 20:55:43,561 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=314 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/B/9acd5806f2cc440f823b467a13248046 2023-04-19 20:55:43,565 DEBUG [Thread-3028] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:43,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:43,569 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:43,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:43,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:43,574 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:43,574 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:43,574 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:43,574 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:43,575 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:43,575 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:43,576 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:43,576 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:43,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:43,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:43,576 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:43,576 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:43,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:43,578 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=314 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/C/e92c9c81e9b04913b595b2510fbf09d6 2023-04-19 20:55:43,578 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:43,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:43,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:43,579 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:43,579 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:55:43,579 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:43,579 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:43,579 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:43,579 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:43,579 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:43,579 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:43,580 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:43,580 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. started... 2023-04-19 20:55:43,580 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. as already flushing 2023-04-19 20:55:43,580 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:43,580 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:43,580 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:43,580 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:43,580 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:43,580 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:43,581 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/A/9a91bef8a8544db6a38a37d8b1eb6d58 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/9a91bef8a8544db6a38a37d8b1eb6d58 2023-04-19 20:55:43,585 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:43,585 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:43,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:43,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:43,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:43,585 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:43,585 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:43,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:43,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:43,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:43,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:43,587 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/9a91bef8a8544db6a38a37d8b1eb6d58, entries=200, sequenceid=314, filesize=14.3 K 2023-04-19 20:55:43,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:43,587 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:43,587 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/B/9acd5806f2cc440f823b467a13248046 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/9acd5806f2cc440f823b467a13248046 2023-04-19 20:55:43,587 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:43,587 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:43,587 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2ac92df0[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:43,589 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:43,589 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:43,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:43,589 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:43,589 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:43,589 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:43,589 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:43,589 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:43,591 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/9acd5806f2cc440f823b467a13248046, entries=150, sequenceid=314, filesize=11.9 K 2023-04-19 20:55:43,592 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:43,592 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:43,592 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:43,592 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/C/e92c9c81e9b04913b595b2510fbf09d6 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/e92c9c81e9b04913b595b2510fbf09d6 2023-04-19 20:55:43,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:43,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:43,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:43,592 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:43,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:43,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:43,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:43,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:43,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:43,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:43,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:43,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:43,595 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:43,595 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:43,596 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:43,596 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:43,596 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:43,596 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:43,596 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/e92c9c81e9b04913b595b2510fbf09d6, entries=150, sequenceid=314, filesize=11.9 K 2023-04-19 20:55:43,597 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 4393734ae917e967cf2b5b4b3b44ed0d in 466ms, sequenceid=314, compaction requested=true 2023-04-19 20:55:43,597 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 4393734ae917e967cf2b5b4b3b44ed0d: 2023-04-19 20:55:43,597 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 20:55:43,597 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-19 20:55:43,597 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-19 20:55:43,597 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 20:55:43,597 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 20:55:43,598 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39122 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-19 20:55:43,598 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1912): 4393734ae917e967cf2b5b4b3b44ed0d/A is initiating minor compaction (all files) 2023-04-19 20:55:43,598 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36682 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-19 20:55:43,598 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 4393734ae917e967cf2b5b4b3b44ed0d/A in TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:43,598 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1912): 4393734ae917e967cf2b5b4b3b44ed0d/B is initiating minor compaction (all files) 2023-04-19 20:55:43,598 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/bdea0565ed934118b46236767cc2487b, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/9ea01a31f7cb415ebbef0aa052832786, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/9a91bef8a8544db6a38a37d8b1eb6d58] into tmpdir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp, totalSize=38.2 K 2023-04-19 20:55:43,598 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4393734ae917e967cf2b5b4b3b44ed0d/B in TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:43,598 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/a86e2147024c4edfad3b8b32a904ebbe, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/01c9a67084494cec85710db4418b8cc3, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/9acd5806f2cc440f823b467a13248046] into tmpdir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp, totalSize=35.8 K 2023-04-19 20:55:43,598 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting bdea0565ed934118b46236767cc2487b, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=143, earliestPutTs=1681937742312 2023-04-19 20:55:43,598 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting a86e2147024c4edfad3b8b32a904ebbe, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=143, earliestPutTs=1681937742312 2023-04-19 20:55:43,599 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 9ea01a31f7cb415ebbef0aa052832786, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=301, earliestPutTs=1681937742892 2023-04-19 20:55:43,599 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 01c9a67084494cec85710db4418b8cc3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=301, earliestPutTs=1681937742892 2023-04-19 20:55:43,599 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 9a91bef8a8544db6a38a37d8b1eb6d58, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=314, earliestPutTs=1681937743122 2023-04-19 20:55:43,599 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 9acd5806f2cc440f823b467a13248046, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=314, earliestPutTs=1681937743122 2023-04-19 20:55:43,599 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:43,599 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:43,599 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:43,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:43,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:43,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:43,599 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:43,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:43,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:43,600 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:43,600 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:43,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:43,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:43,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:43,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:43,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:43,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:43,603 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:43,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:43,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:43,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:43,605 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:43,606 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:43,606 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:43,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:43,606 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:43,606 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:43,606 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:43,606 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:43,606 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:43,606 DEBUG [Thread-3028] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:43,606 DEBUG [Thread-3028] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:43,606 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:43,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:43,606 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:43,606 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:43,606 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:43,606 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:43,606 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:43,607 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] throttle.PressureAwareThroughputController(145): 4393734ae917e967cf2b5b4b3b44ed0d#A#compaction#417 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-19 20:55:43,610 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] throttle.PressureAwareThroughputController(145): 4393734ae917e967cf2b5b4b3b44ed0d#B#compaction#416 average throughput is 1.31 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 20:55:43,671 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-19 20:55:43,706 DEBUG [Thread-3028] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:43,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:43,709 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:43,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:43,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:43,710 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:43,710 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:43,710 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:43,710 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:43,710 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:43,710 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:43,712 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:43,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:43,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:43,712 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:43,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:43,713 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:43,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:43,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:43,713 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:43,714 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:43,714 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:43,714 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:43,714 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:43,714 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:43,714 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:43,714 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:43,714 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:43,714 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:43,714 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. started... 2023-04-19 20:55:43,714 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4393734ae917e967cf2b5b4b3b44ed0d 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-19 20:55:43,715 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4393734ae917e967cf2b5b4b3b44ed0d, store=A 2023-04-19 20:55:43,715 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:55:43,715 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4393734ae917e967cf2b5b4b3b44ed0d, store=B 2023-04-19 20:55:43,715 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:55:43,715 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4393734ae917e967cf2b5b4b3b44ed0d, store=C 2023-04-19 20:55:43,715 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:55:43,741 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=336 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/A/0f1c5c4ed0b14221b9e65af5aaa0e473 2023-04-19 20:55:43,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(9158): Flush requested on 4393734ae917e967cf2b5b4b3b44ed0d 2023-04-19 20:55:43,794 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. as already flushing 2023-04-19 20:55:43,810 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:43,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35492 deadline: 1681937803802, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:43,810 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:43,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35474 deadline: 1681937803803, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:43,811 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:43,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35470 deadline: 1681937803804, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:43,811 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:43,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35490 deadline: 1681937803806, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:43,916 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:43,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35492 deadline: 1681937803911, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:43,916 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:43,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35474 deadline: 1681937803911, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:43,916 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:43,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35470 deadline: 1681937803912, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:43,917 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:43,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35490 deadline: 1681937803912, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:44,027 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/A/3ffc60b4c21d4ccd8c01900796cac8cf as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/3ffc60b4c21d4ccd8c01900796cac8cf 2023-04-19 20:55:44,028 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/B/0e83b450afbb46e79b96d9df35f0f0d3 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/0e83b450afbb46e79b96d9df35f0f0d3 2023-04-19 20:55:44,031 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 4393734ae917e967cf2b5b4b3b44ed0d/A of 4393734ae917e967cf2b5b4b3b44ed0d into 3ffc60b4c21d4ccd8c01900796cac8cf(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 20:55:44,031 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4393734ae917e967cf2b5b4b3b44ed0d: 2023-04-19 20:55:44,032 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d., storeName=4393734ae917e967cf2b5b4b3b44ed0d/A, priority=13, startTime=1681937743597; duration=0sec 2023-04-19 20:55:44,032 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 20:55:44,032 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-19 20:55:44,032 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 4393734ae917e967cf2b5b4b3b44ed0d/B of 4393734ae917e967cf2b5b4b3b44ed0d into 0e83b450afbb46e79b96d9df35f0f0d3(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 20:55:44,032 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4393734ae917e967cf2b5b4b3b44ed0d: 2023-04-19 20:55:44,032 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d., storeName=4393734ae917e967cf2b5b4b3b44ed0d/B, priority=13, startTime=1681937743597; duration=0sec 2023-04-19 20:55:44,033 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 20:55:44,033 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36682 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-19 20:55:44,033 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1912): 4393734ae917e967cf2b5b4b3b44ed0d/C is initiating minor compaction (all files) 2023-04-19 20:55:44,033 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 4393734ae917e967cf2b5b4b3b44ed0d/C in TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:44,033 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/b616a1072bfb42f4928243d32fbd9d65, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/b9e3de9a3ac541f3be7ed560f00da299, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/e92c9c81e9b04913b595b2510fbf09d6] into tmpdir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp, totalSize=35.8 K 2023-04-19 20:55:44,033 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting b616a1072bfb42f4928243d32fbd9d65, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=143, earliestPutTs=1681937742312 2023-04-19 20:55:44,034 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting b9e3de9a3ac541f3be7ed560f00da299, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=301, earliestPutTs=1681937742892 2023-04-19 20:55:44,035 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting e92c9c81e9b04913b595b2510fbf09d6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=314, earliestPutTs=1681937743122 2023-04-19 20:55:44,041 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:44,041 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] throttle.PressureAwareThroughputController(145): 4393734ae917e967cf2b5b4b3b44ed0d#C#compaction#420 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 20:55:44,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35486 deadline: 1681937804036, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:44,121 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:44,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35492 deadline: 1681937804118, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:44,122 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:44,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35490 deadline: 1681937804119, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:44,122 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:44,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35470 deadline: 1681937804119, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:44,122 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:44,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35474 deadline: 1681937804119, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:44,152 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=336 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/B/e6c2e4affed44fd48ab9261a0706d2f4 2023-04-19 20:55:44,162 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=336 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/C/1c4ea4067db14336912112f75758f28c 2023-04-19 20:55:44,166 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/A/0f1c5c4ed0b14221b9e65af5aaa0e473 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/0f1c5c4ed0b14221b9e65af5aaa0e473 2023-04-19 20:55:44,170 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/0f1c5c4ed0b14221b9e65af5aaa0e473, entries=150, sequenceid=336, filesize=11.9 K 2023-04-19 20:55:44,171 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/B/e6c2e4affed44fd48ab9261a0706d2f4 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/e6c2e4affed44fd48ab9261a0706d2f4 2023-04-19 20:55:44,174 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/e6c2e4affed44fd48ab9261a0706d2f4, entries=150, sequenceid=336, filesize=11.9 K 2023-04-19 20:55:44,175 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/C/1c4ea4067db14336912112f75758f28c as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/1c4ea4067db14336912112f75758f28c 2023-04-19 20:55:44,179 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/1c4ea4067db14336912112f75758f28c, entries=150, sequenceid=336, filesize=11.9 K 2023-04-19 20:55:44,180 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 4393734ae917e967cf2b5b4b3b44ed0d in 466ms, sequenceid=336, compaction requested=false 2023-04-19 20:55:44,180 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4393734ae917e967cf2b5b4b3b44ed0d: 2023-04-19 20:55:44,180 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:44,180 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:44,180 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:44,180 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:44,180 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:44,180 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:44,181 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:44,181 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:44,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:44,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:44,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:44,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:44,182 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:44,182 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:44,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:44,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:44,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:44,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:44,183 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:44,183 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@51bd8cd7[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:44,183 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:44,183 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:44,185 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:44,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:44,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:44,185 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:44,185 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:44,185 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:44,185 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:44,185 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:44,186 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:44,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:44,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:44,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:44,186 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:44,187 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:44,187 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:44,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:44,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:44,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:44,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:44,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:44,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:44,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:44,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:44,189 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:44,189 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:44,189 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:44,189 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:44,189 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:44,189 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:44,191 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:44,191 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:44,191 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:44,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:44,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:44,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:44,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:44,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:44,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:44,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:44,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:44,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:44,192 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:44,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:44,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:44,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:44,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:44,195 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:44,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:44,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:44,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:44,197 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:44,197 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:44,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:44,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:44,197 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:44,197 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:44,197 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:44,197 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:44,197 DEBUG [Thread-3028] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:44,197 DEBUG [Thread-3028] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:44,197 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:44,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:44,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:44,198 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:44,198 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:44,198 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:44,198 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:44,198 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:44,297 DEBUG [Thread-3028] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:44,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:44,299 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:44,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:44,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:44,301 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:44,301 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:44,301 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:44,301 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:44,302 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:44,302 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:44,303 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:44,303 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:44,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:44,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:44,303 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:44,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:44,303 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:44,304 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:44,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:44,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:44,304 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:44,304 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:55:44,305 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:44,305 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:44,305 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:44,305 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:44,305 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:44,305 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:44,305 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:44,305 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. started... 2023-04-19 20:55:44,305 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4393734ae917e967cf2b5b4b3b44ed0d 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-19 20:55:44,305 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4393734ae917e967cf2b5b4b3b44ed0d, store=A 2023-04-19 20:55:44,305 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:55:44,306 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4393734ae917e967cf2b5b4b3b44ed0d, store=B 2023-04-19 20:55:44,306 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:55:44,306 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4393734ae917e967cf2b5b4b3b44ed0d, store=C 2023-04-19 20:55:44,306 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:55:44,314 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=350 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/A/15edd9481e0c4eb4a4e6176a7acb7ae8 2023-04-19 20:55:44,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(9158): Flush requested on 4393734ae917e967cf2b5b4b3b44ed0d 2023-04-19 20:55:44,427 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. as already flushing 2023-04-19 20:55:44,450 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/C/add62c469c004655a1cbf2ad2844a86b as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/add62c469c004655a1cbf2ad2844a86b 2023-04-19 20:55:44,454 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 4393734ae917e967cf2b5b4b3b44ed0d/C of 4393734ae917e967cf2b5b4b3b44ed0d into add62c469c004655a1cbf2ad2844a86b(size=12.3 K), total size for store is 24.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 20:55:44,454 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4393734ae917e967cf2b5b4b3b44ed0d: 2023-04-19 20:55:44,454 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d., storeName=4393734ae917e967cf2b5b4b3b44ed0d/C, priority=13, startTime=1681937743597; duration=0sec 2023-04-19 20:55:44,454 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 20:55:44,458 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:44,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35492 deadline: 1681937804451, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:44,458 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:44,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35490 deadline: 1681937804452, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:44,464 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:44,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35474 deadline: 1681937804455, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:44,464 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:44,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35470 deadline: 1681937804456, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:44,561 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:44,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35490 deadline: 1681937804559, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:44,562 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:44,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35492 deadline: 1681937804559, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:44,568 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:44,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35474 deadline: 1681937804565, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:44,568 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:44,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35470 deadline: 1681937804565, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:44,731 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=350 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/B/afe0cadbf1ff4a73a34b4194726fad5a 2023-04-19 20:55:44,746 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=350 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/C/aa28b17809d94b3a996f3c168c7d0799 2023-04-19 20:55:44,751 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/A/15edd9481e0c4eb4a4e6176a7acb7ae8 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/15edd9481e0c4eb4a4e6176a7acb7ae8 2023-04-19 20:55:44,754 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/15edd9481e0c4eb4a4e6176a7acb7ae8, entries=150, sequenceid=350, filesize=11.9 K 2023-04-19 20:55:44,755 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/B/afe0cadbf1ff4a73a34b4194726fad5a as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/afe0cadbf1ff4a73a34b4194726fad5a 2023-04-19 20:55:44,758 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/afe0cadbf1ff4a73a34b4194726fad5a, entries=150, sequenceid=350, filesize=11.9 K 2023-04-19 20:55:44,759 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/C/aa28b17809d94b3a996f3c168c7d0799 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/aa28b17809d94b3a996f3c168c7d0799 2023-04-19 20:55:44,762 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/aa28b17809d94b3a996f3c168c7d0799, entries=150, sequenceid=350, filesize=11.9 K 2023-04-19 20:55:44,763 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 4393734ae917e967cf2b5b4b3b44ed0d in 458ms, sequenceid=350, compaction requested=true 2023-04-19 20:55:44,763 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4393734ae917e967cf2b5b4b3b44ed0d: 2023-04-19 20:55:44,763 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:44,763 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:44,763 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:44,763 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:44,763 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:44,763 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:44,767 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:44,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:44,767 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:44,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:44,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:44,767 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:44,767 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:44,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:44,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:44,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:44,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:44,769 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 4393734ae917e967cf2b5b4b3b44ed0d 3/3 column families, dataSize=120.76 KB heapSize=360.05 KB 2023-04-19 20:55:44,769 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4393734ae917e967cf2b5b4b3b44ed0d, store=A 2023-04-19 20:55:44,769 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 20:55:44,769 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4393734ae917e967cf2b5b4b3b44ed0d, store=B 2023-04-19 20:55:44,769 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 20:55:44,769 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4393734ae917e967cf2b5b4b3b44ed0d, store=C 2023-04-19 20:55:44,770 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 20:55:44,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(9158): Flush requested on 4393734ae917e967cf2b5b4b3b44ed0d 2023-04-19 20:55:44,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:44,771 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:44,771 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6e41c24c[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:44,771 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:44,771 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:44,773 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:44,773 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:44,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:44,773 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:44,773 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:44,773 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:44,774 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:44,774 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:44,775 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:44,775 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:44,775 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:44,775 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:44,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:44,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:44,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:44,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:44,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:44,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:44,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:44,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:44,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:44,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:44,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:44,780 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:44,780 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:44,780 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:44,780 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:44,780 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:44,780 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:44,782 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:44,782 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:44,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:44,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:44,782 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:44,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:44,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:44,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:44,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:44,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:44,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:44,783 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:44,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:44,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:44,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:44,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:44,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:44,784 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:44,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:44,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:44,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:44,787 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:44,787 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:44,787 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:44,787 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:44,787 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:44,787 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:44,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:44,787 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:44,787 DEBUG [Thread-3028] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:44,787 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:44,787 DEBUG [Thread-3028] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:44,787 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:44,787 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:44,787 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:44,787 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:44,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:44,787 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:44,787 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:44,792 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:44,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35490 deadline: 1681937804781, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:44,792 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:44,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35474 deadline: 1681937804782, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:44,793 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:44,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35492 deadline: 1681937804785, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:44,794 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:44,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35470 deadline: 1681937804786, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:44,887 DEBUG [Thread-3028] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:44,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:44,890 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:44,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:44,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:44,891 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:44,891 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:44,891 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:44,891 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:44,891 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:44,892 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:44,894 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:44,894 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:44,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:44,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:44,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:44,895 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:44,895 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:44,895 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:44,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:44,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:44,896 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:44,896 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:55:44,896 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:44,896 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:44,896 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:44,896 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:44,896 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:44,896 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:44,896 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:44,896 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. started... 2023-04-19 20:55:44,896 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. as already flushing 2023-04-19 20:55:44,897 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:44,897 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:44,897 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:44,897 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:44,897 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:44,897 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:44,898 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:44,898 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:44,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:44,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:44,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:44,899 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:44,899 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:44,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:44,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:44,899 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:44,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:44,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35490 deadline: 1681937804893, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:44,900 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:44,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35474 deadline: 1681937804893, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:44,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:44,900 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:44,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35492 deadline: 1681937804894, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:44,900 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:44,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:44,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35470 deadline: 1681937804895, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:44,901 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:44,901 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@559120a0[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:44,901 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:44,901 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:44,903 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:44,903 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:44,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:44,903 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:44,903 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:44,903 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:44,903 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:44,903 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:44,909 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:44,909 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:44,909 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:44,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:44,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:44,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:44,909 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:44,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:44,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:44,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:44,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:44,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:44,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:44,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:44,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:44,911 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:44,911 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:44,912 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:44,912 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:44,912 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:44,912 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:44,913 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:44,913 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:44,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:44,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:44,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:44,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:44,914 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:44,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:44,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:44,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:44,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:44,914 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:44,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:44,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:44,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:44,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:44,915 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:44,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:44,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:44,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:44,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:44,918 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:44,918 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:44,918 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:44,918 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:44,918 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:44,918 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:44,918 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:44,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:44,918 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:44,918 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:55:44,918 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:44,918 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:44,918 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:44,918 DEBUG [Thread-3028] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:44,918 DEBUG [Thread-3028] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:44,918 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:44,918 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:44,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:44,919 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:45,019 DEBUG [Thread-3028] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:45,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:45,021 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:45,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:45,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:45,022 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:45,022 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:45,022 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:45,022 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:45,022 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:45,022 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:45,024 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:45,024 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:45,024 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:45,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:45,025 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:45,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:45,025 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:45,025 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:45,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:45,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:45,025 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:45,026 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:55:45,026 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:45,026 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:45,026 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:45,026 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:45,026 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:45,026 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:45,026 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:45,026 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. started... 2023-04-19 20:55:45,026 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. as already flushing 2023-04-19 20:55:45,026 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:45,026 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:45,026 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:45,026 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:45,026 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:45,026 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:45,028 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:45,028 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:45,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:45,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:45,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:45,029 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:45,029 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:45,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:45,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:45,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:45,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:45,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:45,030 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:45,030 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@36d4314f[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:45,030 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:45,030 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:45,031 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:45,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:45,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:45,031 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:45,031 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:45,031 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:45,032 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:45,032 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:45,033 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:45,033 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:45,033 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:45,033 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:45,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:45,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:45,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:45,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:45,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:45,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:45,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:45,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:45,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:45,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:45,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:45,036 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:45,036 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:45,036 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:45,037 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:45,037 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:45,037 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:45,039 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:45,039 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:45,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:45,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:45,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:45,039 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:45,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:45,039 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:45,039 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:45,039 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:45,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:45,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:45,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:45,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:45,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:45,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:45,042 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:45,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:45,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:45,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:45,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:45,046 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:45,046 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:45,046 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:45,046 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:45,046 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:45,046 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:45,046 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:45,046 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:45,046 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:55:45,046 DEBUG [Thread-3028] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:45,046 DEBUG [Thread-3028] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:45,046 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:45,046 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:45,047 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:45,047 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:45,047 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:45,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:45,047 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:45,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:45,052 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:45,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35486 deadline: 1681937805048, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:45,110 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:45,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35474 deadline: 1681937805101, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:45,110 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:45,110 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:45,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35492 deadline: 1681937805101, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:45,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35470 deadline: 1681937805101, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:45,111 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:45,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35490 deadline: 1681937805101, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:45,146 DEBUG [Thread-3028] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:45,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:45,149 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:45,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:45,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:45,150 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:45,150 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:45,150 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:45,150 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:45,150 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:45,150 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:45,152 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:45,152 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:45,152 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:45,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:45,152 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:45,152 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:45,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:45,152 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:45,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:45,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:45,153 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:45,153 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:55:45,153 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:45,153 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:45,153 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:45,153 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:45,154 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:45,154 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:45,154 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:45,154 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. started... 2023-04-19 20:55:45,154 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. as already flushing 2023-04-19 20:55:45,154 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:45,154 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:45,154 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:45,154 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:45,154 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:45,154 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:45,155 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:45,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:45,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:45,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:45,155 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:45,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:45,156 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:45,156 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:45,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:45,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:45,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:45,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:45,157 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:45,157 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@bb2d876[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:45,157 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:45,157 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:45,163 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:45,164 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:45,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:45,164 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:45,164 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:45,164 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:45,164 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:45,164 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:45,165 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:45,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:45,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:45,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:45,166 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:45,166 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:45,166 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:45,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:45,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:45,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:45,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:45,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:45,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:45,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:45,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:45,168 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:45,168 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:45,168 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:45,168 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:45,168 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:45,168 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:45,171 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:45,171 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:45,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:45,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:45,171 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:45,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:45,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:45,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:45,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:45,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:45,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:45,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:45,173 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:45,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:45,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:45,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:45,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:45,174 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:45,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:45,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:45,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:45,176 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:45,176 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:45,176 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:45,176 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:45,176 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:45,176 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:45,176 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:45,176 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:45,177 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:45,176 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:45,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:45,176 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:45,177 DEBUG [Thread-3028] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:45,177 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:45,177 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:45,177 DEBUG [Thread-3028] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:45,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:45,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:45,179 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=372 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/A/66568d2c0e694ec59e2758f088f94f96 2023-04-19 20:55:45,277 DEBUG [Thread-3028] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:45,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:45,279 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:45,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:45,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:45,281 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:45,281 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:45,281 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:45,281 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:45,281 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:45,281 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:45,283 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:45,283 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:45,283 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:45,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:45,284 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:45,284 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:45,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:45,284 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:45,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:45,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:45,285 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:45,285 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:55:45,285 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:45,285 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:45,285 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:45,285 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:45,285 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:45,285 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:45,285 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:45,285 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. started... 2023-04-19 20:55:45,286 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. as already flushing 2023-04-19 20:55:45,286 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:45,286 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:45,286 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:45,286 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:45,286 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:45,286 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:45,287 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:45,287 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:45,287 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:45,288 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:45,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:45,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:45,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:45,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:45,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:45,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:45,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:45,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:45,290 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:45,290 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6b0138ee[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:45,290 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:45,290 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:45,291 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:45,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:45,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:45,291 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:45,291 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:45,291 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:45,291 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:45,291 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:45,292 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:45,293 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:45,293 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:45,293 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:45,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:45,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:45,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:45,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:45,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:45,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:45,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:45,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:45,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:45,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:45,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:45,295 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:45,295 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:45,295 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:45,295 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:45,295 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:45,295 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:45,300 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:45,300 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:45,300 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:45,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:45,300 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:45,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:45,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:45,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:45,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:45,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:45,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:45,301 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:45,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:45,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:45,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:45,303 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:45,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:45,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:45,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:45,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:45,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:45,305 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:45,305 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:45,305 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:45,305 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:45,305 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:45,305 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:45,305 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:45,306 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:45,306 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:45,306 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:45,306 DEBUG [Thread-3028] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:45,306 DEBUG [Thread-3028] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:45,306 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:55:45,306 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:45,306 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:45,306 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:45,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:45,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:45,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:45,406 DEBUG [Thread-3028] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:45,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:45,408 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:45,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:45,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:45,409 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:45,409 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:45,409 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:45,409 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:45,410 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:45,410 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:45,411 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:45,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:45,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:45,412 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:45,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:45,412 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:45,412 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:45,412 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:45,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:45,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:45,413 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:45,413 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:55:45,413 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:45,413 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:45,413 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:45,413 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:45,413 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:45,413 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:45,413 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:45,413 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. started... 2023-04-19 20:55:45,414 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. as already flushing 2023-04-19 20:55:45,414 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:45,414 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:45,414 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:45,414 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:45,414 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:45,414 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:45,416 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:45,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35474 deadline: 1681937805411, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:45,417 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:45,417 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:45,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:45,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:45,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:45,417 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:45,417 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:45,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:45,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:45,418 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:45,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:45,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35490 deadline: 1681937805412, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:45,418 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:45,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35470 deadline: 1681937805412, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:45,420 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:45,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:45,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35492 deadline: 1681937805414, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:45,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:45,420 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:45,420 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@37c9810e[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:45,420 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:45,421 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:45,422 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:45,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:45,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:45,422 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:45,422 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:45,422 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:45,422 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:45,422 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:45,423 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:45,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:45,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:45,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:45,423 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:45,424 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:45,424 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:45,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:45,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:45,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:45,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:45,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:45,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:45,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:45,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:45,426 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:45,426 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:45,426 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:45,426 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:45,426 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:45,426 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:45,428 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:45,428 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:45,428 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:45,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:45,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:45,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:45,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:45,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:45,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:45,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:45,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:45,428 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:45,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:45,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:45,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:45,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:45,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:45,430 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:45,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:45,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:45,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:45,433 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:45,433 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:45,433 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:45,433 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:45,433 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:45,433 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:45,433 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:45,433 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:45,433 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:45,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:45,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:45,433 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:55:45,434 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:45,434 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:45,434 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:45,434 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:45,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:45,434 DEBUG [Thread-3028] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:45,435 DEBUG [Thread-3028] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:45,535 DEBUG [Thread-3028] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:45,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:45,537 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:45,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:45,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:45,538 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:45,538 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:45,538 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:45,538 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:45,539 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:45,539 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:45,540 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:45,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:45,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:45,540 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:45,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:45,541 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:45,541 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:45,541 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:45,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:45,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:45,541 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:45,542 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:55:45,542 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:45,542 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:45,542 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:45,542 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:45,542 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:45,542 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:45,542 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:45,542 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. started... 2023-04-19 20:55:45,542 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. as already flushing 2023-04-19 20:55:45,542 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:45,542 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:45,542 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:45,543 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:45,543 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:45,543 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:45,544 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:45,544 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:45,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:45,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:45,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:45,545 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:45,545 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:45,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:45,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:45,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:45,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:45,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:45,546 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:45,546 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@200090f1[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:45,546 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:45,546 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:45,548 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:45,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:45,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:45,548 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:45,548 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:45,548 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:45,548 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:45,548 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:45,551 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:45,551 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:45,551 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:45,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:45,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:45,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:45,551 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:45,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:45,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:45,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:45,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:45,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:45,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:45,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:45,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:45,554 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:45,554 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:45,554 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:45,554 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:45,554 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:45,554 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:45,556 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:45,556 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:45,556 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:45,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:45,556 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:45,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:45,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:45,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:45,556 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:45,556 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:45,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:45,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:45,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:45,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:45,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:45,557 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:45,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:45,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:45,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:45,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:45,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:45,560 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:45,560 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:45,560 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:45,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:45,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:45,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:45,561 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:45,561 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:45,561 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:45,562 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:45,562 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:45,562 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:45,562 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:45,562 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:45,562 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:45,562 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:45,562 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:55:45,562 DEBUG [Thread-3028] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:45,562 DEBUG [Thread-3028] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:45,591 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=372 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/B/a3330176b169407a981620c68915d582 2023-04-19 20:55:45,603 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=372 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/C/c543956a42754d7aa0512223ea0af147 2023-04-19 20:55:45,606 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/A/66568d2c0e694ec59e2758f088f94f96 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/66568d2c0e694ec59e2758f088f94f96 2023-04-19 20:55:45,610 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/66568d2c0e694ec59e2758f088f94f96, entries=200, sequenceid=372, filesize=14.3 K 2023-04-19 20:55:45,611 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/B/a3330176b169407a981620c68915d582 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/a3330176b169407a981620c68915d582 2023-04-19 20:55:45,614 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/a3330176b169407a981620c68915d582, entries=150, sequenceid=372, filesize=11.9 K 2023-04-19 20:55:45,615 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/C/c543956a42754d7aa0512223ea0af147 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/c543956a42754d7aa0512223ea0af147 2023-04-19 20:55:45,618 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/c543956a42754d7aa0512223ea0af147, entries=150, sequenceid=372, filesize=11.9 K 2023-04-19 20:55:45,619 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~360 KB/368640, currentSize=53.67 KB/54960 for 4393734ae917e967cf2b5b4b3b44ed0d in 850ms, sequenceid=372, compaction requested=true 2023-04-19 20:55:45,619 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 4393734ae917e967cf2b5b4b3b44ed0d: 2023-04-19 20:55:45,619 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 20:55:45,619 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-19 20:55:45,619 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-19 20:55:45,619 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 20:55:45,619 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 20:55:45,626 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 51560 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-19 20:55:45,626 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1912): 4393734ae917e967cf2b5b4b3b44ed0d/A is initiating minor compaction (all files) 2023-04-19 20:55:45,626 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49120 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-19 20:55:45,626 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 4393734ae917e967cf2b5b4b3b44ed0d/A in TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:45,626 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1912): 4393734ae917e967cf2b5b4b3b44ed0d/B is initiating minor compaction (all files) 2023-04-19 20:55:45,626 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4393734ae917e967cf2b5b4b3b44ed0d/B in TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:45,626 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/3ffc60b4c21d4ccd8c01900796cac8cf, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/0f1c5c4ed0b14221b9e65af5aaa0e473, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/15edd9481e0c4eb4a4e6176a7acb7ae8, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/66568d2c0e694ec59e2758f088f94f96] into tmpdir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp, totalSize=50.4 K 2023-04-19 20:55:45,626 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/0e83b450afbb46e79b96d9df35f0f0d3, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/e6c2e4affed44fd48ab9261a0706d2f4, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/afe0cadbf1ff4a73a34b4194726fad5a, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/a3330176b169407a981620c68915d582] into tmpdir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp, totalSize=48.0 K 2023-04-19 20:55:45,627 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 3ffc60b4c21d4ccd8c01900796cac8cf, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=314, earliestPutTs=1681937743122 2023-04-19 20:55:45,627 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 0e83b450afbb46e79b96d9df35f0f0d3, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=314, earliestPutTs=1681937743122 2023-04-19 20:55:45,627 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 0f1c5c4ed0b14221b9e65af5aaa0e473, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=336, earliestPutTs=1681937743158 2023-04-19 20:55:45,627 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting e6c2e4affed44fd48ab9261a0706d2f4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=336, earliestPutTs=1681937743158 2023-04-19 20:55:45,628 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 15edd9481e0c4eb4a4e6176a7acb7ae8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=350, earliestPutTs=1681937743801 2023-04-19 20:55:45,628 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting afe0cadbf1ff4a73a34b4194726fad5a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=350, earliestPutTs=1681937743801 2023-04-19 20:55:45,628 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 66568d2c0e694ec59e2758f088f94f96, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=372, earliestPutTs=1681937744446 2023-04-19 20:55:45,628 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting a3330176b169407a981620c68915d582, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=372, earliestPutTs=1681937744446 2023-04-19 20:55:45,637 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] throttle.PressureAwareThroughputController(145): 4393734ae917e967cf2b5b4b3b44ed0d#A#compaction#428 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 20:55:45,637 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] throttle.PressureAwareThroughputController(145): 4393734ae917e967cf2b5b4b3b44ed0d#B#compaction#429 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 20:55:45,662 DEBUG [Thread-3028] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:45,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:45,665 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:45,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:45,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:45,667 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:45,668 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:45,668 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:45,668 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:45,668 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/B/472e8d25dc514814b624e1d57ed53ec6 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/472e8d25dc514814b624e1d57ed53ec6 2023-04-19 20:55:45,669 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:45,669 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:45,670 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/A/785eec2bf2554d30b5c16f9a25379568 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/785eec2bf2554d30b5c16f9a25379568 2023-04-19 20:55:45,670 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:45,670 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:45,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:45,671 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:45,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:45,671 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:45,671 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:45,671 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:45,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:45,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:45,672 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:45,672 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:55:45,672 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:45,672 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:45,672 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:45,672 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:45,675 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 4393734ae917e967cf2b5b4b3b44ed0d/B of 4393734ae917e967cf2b5b4b3b44ed0d into 472e8d25dc514814b624e1d57ed53ec6(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 20:55:45,675 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4393734ae917e967cf2b5b4b3b44ed0d: 2023-04-19 20:55:45,675 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d., storeName=4393734ae917e967cf2b5b4b3b44ed0d/B, priority=12, startTime=1681937745619; duration=0sec 2023-04-19 20:55:45,675 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 20:55:45,675 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-19 20:55:45,676 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:45,676 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:45,676 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49120 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-19 20:55:45,676 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:45,676 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1912): 4393734ae917e967cf2b5b4b3b44ed0d/C is initiating minor compaction (all files) 2023-04-19 20:55:45,676 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. started... 2023-04-19 20:55:45,676 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4393734ae917e967cf2b5b4b3b44ed0d/C in TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:45,676 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/add62c469c004655a1cbf2ad2844a86b, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/1c4ea4067db14336912112f75758f28c, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/aa28b17809d94b3a996f3c168c7d0799, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/c543956a42754d7aa0512223ea0af147] into tmpdir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp, totalSize=48.0 K 2023-04-19 20:55:45,676 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4393734ae917e967cf2b5b4b3b44ed0d 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-19 20:55:45,676 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4393734ae917e967cf2b5b4b3b44ed0d, store=A 2023-04-19 20:55:45,677 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:55:45,677 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4393734ae917e967cf2b5b4b3b44ed0d, store=B 2023-04-19 20:55:45,677 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:55:45,677 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting add62c469c004655a1cbf2ad2844a86b, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=314, earliestPutTs=1681937743122 2023-04-19 20:55:45,677 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4393734ae917e967cf2b5b4b3b44ed0d, store=C 2023-04-19 20:55:45,677 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:55:45,678 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 1c4ea4067db14336912112f75758f28c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=336, earliestPutTs=1681937743158 2023-04-19 20:55:45,678 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting aa28b17809d94b3a996f3c168c7d0799, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=350, earliestPutTs=1681937743801 2023-04-19 20:55:45,678 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting c543956a42754d7aa0512223ea0af147, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=372, earliestPutTs=1681937744446 2023-04-19 20:55:45,680 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 4393734ae917e967cf2b5b4b3b44ed0d/A of 4393734ae917e967cf2b5b4b3b44ed0d into 785eec2bf2554d30b5c16f9a25379568(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 20:55:45,680 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4393734ae917e967cf2b5b4b3b44ed0d: 2023-04-19 20:55:45,680 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d., storeName=4393734ae917e967cf2b5b4b3b44ed0d/A, priority=12, startTime=1681937745619; duration=0sec 2023-04-19 20:55:45,680 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 20:55:45,702 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] throttle.PressureAwareThroughputController(145): 4393734ae917e967cf2b5b4b3b44ed0d#C#compaction#431 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 20:55:45,712 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=384 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/A/46d649f1d4c64cbbaad84dd81d645bac 2023-04-19 20:55:45,728 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/C/e3b17b126f24483083b3bd2d89935385 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/e3b17b126f24483083b3bd2d89935385 2023-04-19 20:55:45,733 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 4393734ae917e967cf2b5b4b3b44ed0d/C of 4393734ae917e967cf2b5b4b3b44ed0d into e3b17b126f24483083b3bd2d89935385(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 20:55:45,733 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4393734ae917e967cf2b5b4b3b44ed0d: 2023-04-19 20:55:45,733 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d., storeName=4393734ae917e967cf2b5b4b3b44ed0d/C, priority=12, startTime=1681937745619; duration=0sec 2023-04-19 20:55:45,733 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 20:55:45,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(9158): Flush requested on 4393734ae917e967cf2b5b4b3b44ed0d 2023-04-19 20:55:45,923 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. as already flushing 2023-04-19 20:55:45,946 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:45,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35474 deadline: 1681937805939, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:45,947 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681937745935, max timestamp=1681937745945Num uniques 859; 2023-04-19 20:55:45,947 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-19 20:55:45,948 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681937745935, max timestamp=1681937745945Num uniques 859; 2023-04-19 20:55:45,948 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681937745935, max timestamp=1681937745945Num uniques 859; 2023-04-19 20:55:45,948 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-19 20:55:45,948 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-19 20:55:45,990 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:45,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35492 deadline: 1681937805987, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:45,998 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:45,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35470 deadline: 1681937805988, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:45,999 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:45,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35490 deadline: 1681937805988, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:46,054 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:46,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35474 deadline: 1681937806049, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:46,095 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:46,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35492 deadline: 1681937806092, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:46,105 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:46,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35470 deadline: 1681937806100, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:46,105 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:46,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35490 deadline: 1681937806100, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:46,131 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=384 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/B/2fdbdee3b0c04d0f8490ceb791ca32ab 2023-04-19 20:55:46,164 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=384 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/C/9f1c8421794945c6ac876bb703cffcfb 2023-04-19 20:55:46,169 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/A/46d649f1d4c64cbbaad84dd81d645bac as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/46d649f1d4c64cbbaad84dd81d645bac 2023-04-19 20:55:46,175 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/46d649f1d4c64cbbaad84dd81d645bac, entries=150, sequenceid=384, filesize=11.9 K 2023-04-19 20:55:46,176 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/B/2fdbdee3b0c04d0f8490ceb791ca32ab as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/2fdbdee3b0c04d0f8490ceb791ca32ab 2023-04-19 20:55:46,179 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/2fdbdee3b0c04d0f8490ceb791ca32ab, entries=150, sequenceid=384, filesize=11.9 K 2023-04-19 20:55:46,180 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/C/9f1c8421794945c6ac876bb703cffcfb as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/9f1c8421794945c6ac876bb703cffcfb 2023-04-19 20:55:46,186 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/9f1c8421794945c6ac876bb703cffcfb, entries=150, sequenceid=384, filesize=11.9 K 2023-04-19 20:55:46,187 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=134.18 KB/137400 for 4393734ae917e967cf2b5b4b3b44ed0d in 510ms, sequenceid=384, compaction requested=false 2023-04-19 20:55:46,187 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4393734ae917e967cf2b5b4b3b44ed0d: 2023-04-19 20:55:46,187 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:46,187 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:46,187 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:46,187 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:46,187 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:46,187 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:46,193 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:46,193 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:46,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:46,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:46,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:46,193 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:46,193 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:46,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:46,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:46,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:46,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:46,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:46,198 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:46,198 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3d6c9def[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:46,198 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:46,198 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:46,199 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:46,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:46,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:46,200 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:46,200 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:46,200 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:46,200 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:46,200 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:46,201 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:46,201 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:46,201 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:46,202 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:46,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:46,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:46,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:46,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:46,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:46,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:46,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:46,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:46,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:46,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:46,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:46,206 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:46,206 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:46,206 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:46,206 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:46,206 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:46,206 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:46,209 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:46,209 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:46,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:46,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:46,210 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:46,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:46,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:46,210 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:46,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:46,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:46,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:46,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:46,211 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:46,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:46,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:46,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:46,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:46,213 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:46,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:46,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:46,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:46,219 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:46,219 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:46,220 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:46,220 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:46,220 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:46,220 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:46,220 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:46,220 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:46,220 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:46,220 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:46,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:46,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:46,220 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:46,220 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:46,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:46,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:46,220 DEBUG [Thread-3028] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:46,220 DEBUG [Thread-3028] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:46,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(9158): Flush requested on 4393734ae917e967cf2b5b4b3b44ed0d 2023-04-19 20:55:46,260 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 4393734ae917e967cf2b5b4b3b44ed0d 3/3 column families, dataSize=140.89 KB heapSize=395.53 KB 2023-04-19 20:55:46,260 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4393734ae917e967cf2b5b4b3b44ed0d, store=A 2023-04-19 20:55:46,260 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 20:55:46,260 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4393734ae917e967cf2b5b4b3b44ed0d, store=B 2023-04-19 20:55:46,260 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 20:55:46,260 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4393734ae917e967cf2b5b4b3b44ed0d, store=C 2023-04-19 20:55:46,261 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 20:55:46,269 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=422 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/A/83f1252ec5f44d1585561d734723ea22 2023-04-19 20:55:46,279 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=422 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/B/22e2a025b60b4298bcb1fb60083eb171 2023-04-19 20:55:46,289 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=422 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/C/eb59f8cd86e84e779614aeb9b2dad22e 2023-04-19 20:55:46,298 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/A/83f1252ec5f44d1585561d734723ea22 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/83f1252ec5f44d1585561d734723ea22 2023-04-19 20:55:46,302 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/83f1252ec5f44d1585561d734723ea22, entries=200, sequenceid=422, filesize=14.3 K 2023-04-19 20:55:46,302 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/B/22e2a025b60b4298bcb1fb60083eb171 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/22e2a025b60b4298bcb1fb60083eb171 2023-04-19 20:55:46,305 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:46,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35492 deadline: 1681937806297, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:46,305 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/22e2a025b60b4298bcb1fb60083eb171, entries=150, sequenceid=422, filesize=11.9 K 2023-04-19 20:55:46,306 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/C/eb59f8cd86e84e779614aeb9b2dad22e as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/eb59f8cd86e84e779614aeb9b2dad22e 2023-04-19 20:55:46,311 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:46,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35474 deadline: 1681937806302, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:46,311 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:46,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35490 deadline: 1681937806306, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:46,311 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:46,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35470 deadline: 1681937806306, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:46,315 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/eb59f8cd86e84e779614aeb9b2dad22e, entries=150, sequenceid=422, filesize=11.9 K 2023-04-19 20:55:46,316 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~395.48 KB/404976, currentSize=40.25 KB/41220 for 4393734ae917e967cf2b5b4b3b44ed0d in 56ms, sequenceid=422, compaction requested=true 2023-04-19 20:55:46,316 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 4393734ae917e967cf2b5b4b3b44ed0d: 2023-04-19 20:55:46,316 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 20:55:46,316 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 20:55:46,316 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-19 20:55:46,316 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-19 20:55:46,316 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-19 20:55:46,318 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39510 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-19 20:55:46,318 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1912): 4393734ae917e967cf2b5b4b3b44ed0d/A is initiating minor compaction (all files) 2023-04-19 20:55:46,318 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 4393734ae917e967cf2b5b4b3b44ed0d/A in TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:46,318 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/785eec2bf2554d30b5c16f9a25379568, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/46d649f1d4c64cbbaad84dd81d645bac, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/83f1252ec5f44d1585561d734723ea22] into tmpdir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp, totalSize=38.6 K 2023-04-19 20:55:46,318 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 785eec2bf2554d30b5c16f9a25379568, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=372, earliestPutTs=1681937744446 2023-04-19 20:55:46,318 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37070 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-19 20:55:46,319 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1912): 4393734ae917e967cf2b5b4b3b44ed0d/B is initiating minor compaction (all files) 2023-04-19 20:55:46,319 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4393734ae917e967cf2b5b4b3b44ed0d/B in TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:46,319 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/472e8d25dc514814b624e1d57ed53ec6, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/2fdbdee3b0c04d0f8490ceb791ca32ab, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/22e2a025b60b4298bcb1fb60083eb171] into tmpdir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp, totalSize=36.2 K 2023-04-19 20:55:46,319 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 46d649f1d4c64cbbaad84dd81d645bac, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=384, earliestPutTs=1681937744779 2023-04-19 20:55:46,319 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 472e8d25dc514814b624e1d57ed53ec6, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=372, earliestPutTs=1681937744446 2023-04-19 20:55:46,320 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 83f1252ec5f44d1585561d734723ea22, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=422, earliestPutTs=1681937745979 2023-04-19 20:55:46,320 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 2fdbdee3b0c04d0f8490ceb791ca32ab, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=384, earliestPutTs=1681937744779 2023-04-19 20:55:46,321 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 22e2a025b60b4298bcb1fb60083eb171, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=422, earliestPutTs=1681937745986 2023-04-19 20:55:46,321 DEBUG [Thread-3028] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:46,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:46,323 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:46,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:46,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:46,328 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:46,328 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:46,328 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:46,328 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:46,328 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:46,328 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:46,330 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:46,330 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:46,330 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:46,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:46,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:46,330 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:46,330 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:46,330 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:46,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:46,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:46,331 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:46,337 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] throttle.PressureAwareThroughputController(145): 4393734ae917e967cf2b5b4b3b44ed0d#A#compaction#437 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 20:55:46,345 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] throttle.PressureAwareThroughputController(145): 4393734ae917e967cf2b5b4b3b44ed0d#B#compaction#438 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 20:55:46,349 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:55:46,349 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:46,349 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:46,349 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:46,349 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:46,352 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:46,352 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:46,353 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:46,353 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. started... 2023-04-19 20:55:46,353 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4393734ae917e967cf2b5b4b3b44ed0d 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-19 20:55:46,353 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4393734ae917e967cf2b5b4b3b44ed0d, store=A 2023-04-19 20:55:46,353 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:55:46,353 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4393734ae917e967cf2b5b4b3b44ed0d, store=B 2023-04-19 20:55:46,353 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:55:46,353 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4393734ae917e967cf2b5b4b3b44ed0d, store=C 2023-04-19 20:55:46,353 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:55:46,397 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/B/b25198594675479583e0b463c8587b1c as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/b25198594675479583e0b463c8587b1c 2023-04-19 20:55:46,402 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 4393734ae917e967cf2b5b4b3b44ed0d/B of 4393734ae917e967cf2b5b4b3b44ed0d into b25198594675479583e0b463c8587b1c(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 20:55:46,402 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4393734ae917e967cf2b5b4b3b44ed0d: 2023-04-19 20:55:46,402 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d., storeName=4393734ae917e967cf2b5b4b3b44ed0d/B, priority=13, startTime=1681937746316; duration=0sec 2023-04-19 20:55:46,402 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 20:55:46,402 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-19 20:55:46,403 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37070 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-19 20:55:46,403 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1912): 4393734ae917e967cf2b5b4b3b44ed0d/C is initiating minor compaction (all files) 2023-04-19 20:55:46,403 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4393734ae917e967cf2b5b4b3b44ed0d/C in TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:46,403 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/e3b17b126f24483083b3bd2d89935385, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/9f1c8421794945c6ac876bb703cffcfb, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/eb59f8cd86e84e779614aeb9b2dad22e] into tmpdir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp, totalSize=36.2 K 2023-04-19 20:55:46,404 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting e3b17b126f24483083b3bd2d89935385, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=372, earliestPutTs=1681937744446 2023-04-19 20:55:46,404 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 9f1c8421794945c6ac876bb703cffcfb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=384, earliestPutTs=1681937744779 2023-04-19 20:55:46,404 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting eb59f8cd86e84e779614aeb9b2dad22e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=422, earliestPutTs=1681937745986 2023-04-19 20:55:46,409 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] throttle.PressureAwareThroughputController(145): 4393734ae917e967cf2b5b4b3b44ed0d#C#compaction#440 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 20:55:46,418 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/C/717cf7885aff481499668309cb0affbc as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/717cf7885aff481499668309cb0affbc 2023-04-19 20:55:46,426 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 4393734ae917e967cf2b5b4b3b44ed0d/C of 4393734ae917e967cf2b5b4b3b44ed0d into 717cf7885aff481499668309cb0affbc(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 20:55:46,426 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4393734ae917e967cf2b5b4b3b44ed0d: 2023-04-19 20:55:46,426 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d., storeName=4393734ae917e967cf2b5b4b3b44ed0d/C, priority=13, startTime=1681937746316; duration=0sec 2023-04-19 20:55:46,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(9158): Flush requested on 4393734ae917e967cf2b5b4b3b44ed0d 2023-04-19 20:55:46,426 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 20:55:46,426 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. as already flushing 2023-04-19 20:55:46,514 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681937746475, max timestamp=1681937746513Num uniques 859; 2023-04-19 20:55:46,514 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 20:55:46,514 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681937746475, max timestamp=1681937746513Num uniques 859; 2023-04-19 20:55:46,515 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 20:55:46,515 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681937746475, max timestamp=1681937746513Num uniques 859; 2023-04-19 20:55:46,515 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 20:55:46,620 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681937746602, max timestamp=1681937746618Num uniques 1018; 2023-04-19 20:55:46,620 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-19 20:55:46,621 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681937746602, max timestamp=1681937746618Num uniques 1018; 2023-04-19 20:55:46,621 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-19 20:55:46,621 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681937746602, max timestamp=1681937746618Num uniques 1018; 2023-04-19 20:55:46,621 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-19 20:55:46,652 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:46,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35470 deadline: 1681937806650, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:46,653 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1681937746642, max timestamp=1681937746651Num uniques 1027; 2023-04-19 20:55:46,653 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10392, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-04-19 20:55:46,653 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1681937746642, max timestamp=1681937746651Num uniques 1027; 2023-04-19 20:55:46,654 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10392, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-04-19 20:55:46,654 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1681937746642, max timestamp=1681937746651Num uniques 1027; 2023-04-19 20:55:46,654 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10392, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-04-19 20:55:46,707 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:46,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35492 deadline: 1681937806698, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:46,707 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:46,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35490 deadline: 1681937806698, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:46,707 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:46,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35474 deadline: 1681937806705, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:46,760 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:46,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35470 deadline: 1681937806753, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:46,787 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/A/476f2dffd0a0400d8b8e9e7891445841 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/476f2dffd0a0400d8b8e9e7891445841 2023-04-19 20:55:46,790 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 4393734ae917e967cf2b5b4b3b44ed0d/A of 4393734ae917e967cf2b5b4b3b44ed0d into 476f2dffd0a0400d8b8e9e7891445841(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 20:55:46,790 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4393734ae917e967cf2b5b4b3b44ed0d: 2023-04-19 20:55:46,790 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d., storeName=4393734ae917e967cf2b5b4b3b44ed0d/A, priority=13, startTime=1681937746316; duration=0sec 2023-04-19 20:55:46,791 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 20:55:46,802 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=431 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/A/3d713cd6cad44bf8909494a274ee3256 2023-04-19 20:55:46,812 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=431 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/B/9a2493853ea342f7b2bff95de1f09376 2023-04-19 20:55:46,817 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:46,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35490 deadline: 1681937806809, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:46,817 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:46,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35492 deadline: 1681937806809, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:46,818 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:46,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35474 deadline: 1681937806809, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:46,965 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:46,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35470 deadline: 1681937806961, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:47,021 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:47,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35474 deadline: 1681937807019, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:47,022 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:47,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35490 deadline: 1681937807019, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:47,024 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:47,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35492 deadline: 1681937807020, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:47,078 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:47,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35486 deadline: 1681937807071, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:47,080 DEBUG [Thread-3026] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4177 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d., hostname=jenkins-hbase4.apache.org,35247,1681937616971, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-19 20:55:47,224 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=431 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/C/8aab4a235bb34537849c7f92acaa9ac0 2023-04-19 20:55:47,230 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/A/3d713cd6cad44bf8909494a274ee3256 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/3d713cd6cad44bf8909494a274ee3256 2023-04-19 20:55:47,234 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/3d713cd6cad44bf8909494a274ee3256, entries=150, sequenceid=431, filesize=11.9 K 2023-04-19 20:55:47,235 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/B/9a2493853ea342f7b2bff95de1f09376 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/9a2493853ea342f7b2bff95de1f09376 2023-04-19 20:55:47,238 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/9a2493853ea342f7b2bff95de1f09376, entries=150, sequenceid=431, filesize=11.9 K 2023-04-19 20:55:47,239 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/C/8aab4a235bb34537849c7f92acaa9ac0 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/8aab4a235bb34537849c7f92acaa9ac0 2023-04-19 20:55:47,242 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/8aab4a235bb34537849c7f92acaa9ac0, entries=150, sequenceid=431, filesize=11.9 K 2023-04-19 20:55:47,243 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=140.89 KB/144270 for 4393734ae917e967cf2b5b4b3b44ed0d in 890ms, sequenceid=431, compaction requested=false 2023-04-19 20:55:47,243 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4393734ae917e967cf2b5b4b3b44ed0d: 2023-04-19 20:55:47,243 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:47,243 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:47,243 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:47,243 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:47,243 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:47,243 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:47,246 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:47,246 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:47,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:47,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:47,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:47,247 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:47,247 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:47,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:47,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:47,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:47,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:47,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:47,248 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:47,248 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5fc804e4[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:47,248 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:47,248 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:47,251 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:47,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:47,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:47,251 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:47,251 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:47,251 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:47,251 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:47,251 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:47,253 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:47,253 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:47,253 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:47,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:47,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:47,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:47,253 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:47,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:47,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:47,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:47,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:47,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:47,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:47,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:47,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:47,255 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:47,255 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:47,255 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:47,256 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:47,256 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:47,256 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:47,257 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:47,257 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:47,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:47,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:47,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:47,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:47,258 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:47,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:47,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:47,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:47,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:47,259 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:47,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:47,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:47,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:47,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:47,260 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:47,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:47,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:47,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:47,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:47,263 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:47,263 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:47,263 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:47,263 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:47,263 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:47,263 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:47,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:47,263 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:47,263 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:47,263 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:47,264 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:47,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:47,263 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:47,264 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:47,264 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:55:47,264 DEBUG [Thread-3028] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:47,264 DEBUG [Thread-3028] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:47,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:47,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:47,273 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 4393734ae917e967cf2b5b4b3b44ed0d 3/3 column families, dataSize=147.60 KB heapSize=419.16 KB 2023-04-19 20:55:47,273 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4393734ae917e967cf2b5b4b3b44ed0d, store=A 2023-04-19 20:55:47,274 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-19 20:55:47,274 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4393734ae917e967cf2b5b4b3b44ed0d, store=B 2023-04-19 20:55:47,274 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-19 20:55:47,274 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4393734ae917e967cf2b5b4b3b44ed0d, store=C 2023-04-19 20:55:47,274 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-19 20:55:47,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(9158): Flush requested on 4393734ae917e967cf2b5b4b3b44ed0d 2023-04-19 20:55:47,301 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:47,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35470 deadline: 1681937807295, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:47,326 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:47,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35490 deadline: 1681937807323, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:47,329 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:47,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35474 deadline: 1681937807323, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:47,332 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:47,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35492 deadline: 1681937807325, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:47,364 DEBUG [Thread-3028] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:47,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:47,366 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:47,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:47,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:47,367 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:47,367 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:47,367 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:47,367 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:47,368 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:47,368 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:47,370 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:47,370 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:47,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:47,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:47,370 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:47,370 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:47,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:47,371 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:47,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:47,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:47,371 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:47,371 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:55:47,371 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:47,371 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:47,371 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:47,371 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:47,372 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:47,372 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:47,372 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:47,372 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. started... 2023-04-19 20:55:47,372 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. as already flushing 2023-04-19 20:55:47,372 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:47,372 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:47,372 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:47,372 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:47,372 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:47,372 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:47,373 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:47,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:47,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:47,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:47,373 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:47,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:47,374 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:47,374 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:47,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:47,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:47,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:47,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:47,375 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:47,375 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@24722fe2[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:47,375 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:47,375 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:47,376 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:47,376 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:47,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:47,376 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:47,376 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:47,376 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:47,376 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:47,376 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:47,378 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:47,378 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:47,378 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:47,378 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:47,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:47,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:47,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:47,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:47,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:47,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:47,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:47,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:47,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:47,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:47,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:47,380 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:47,380 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:47,380 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:47,380 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:47,380 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:47,380 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:47,383 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:47,383 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:47,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:47,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:47,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:47,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:47,383 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:47,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:47,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:47,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:47,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:47,383 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:47,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:47,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:47,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:47,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:47,384 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:47,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:47,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:47,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:47,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:47,386 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:47,386 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:47,386 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:47,386 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:47,386 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:47,386 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:47,386 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:47,386 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:47,386 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:47,386 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:47,386 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:47,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:47,386 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:47,387 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:47,387 DEBUG [Thread-3028] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:47,387 DEBUG [Thread-3028] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:47,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:47,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:47,412 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:47,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35470 deadline: 1681937807402, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:47,487 DEBUG [Thread-3028] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:47,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:47,489 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:47,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:47,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:47,490 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:47,490 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:47,490 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:47,490 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:47,491 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:47,491 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:47,492 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:47,492 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:47,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:47,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:47,493 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:47,493 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:47,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:47,493 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:47,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:47,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:47,494 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:47,494 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:55:47,494 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:47,494 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:47,494 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:47,494 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:47,494 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:47,494 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:47,494 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:47,494 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. started... 2023-04-19 20:55:47,494 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. as already flushing 2023-04-19 20:55:47,495 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:47,495 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:47,495 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:47,495 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:47,495 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:47,495 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:47,496 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:47,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:47,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:47,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:47,498 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:47,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:47,498 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:47,498 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:47,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:47,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:47,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:47,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:47,500 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:47,500 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@51a485[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:47,500 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:47,500 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:47,502 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:47,502 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:47,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:47,502 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:47,502 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:47,502 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:47,502 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:47,502 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:47,503 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:47,503 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:47,503 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:47,503 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:47,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:47,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:47,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:47,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:47,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:47,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:47,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:47,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:47,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:47,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:47,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:47,506 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:47,506 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:47,506 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:47,507 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:47,507 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:47,507 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:47,508 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:47,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:47,508 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:47,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:47,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:47,508 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:47,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:47,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:47,509 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:47,509 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:47,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:47,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:47,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:47,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:47,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:47,510 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:47,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:47,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:47,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:47,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:47,512 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:47,512 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:47,512 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:47,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:47,512 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:47,512 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:47,512 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:47,513 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:47,513 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:47,513 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:47,513 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:47,513 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:47,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:47,512 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:47,513 DEBUG [Thread-3028] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:47,513 DEBUG [Thread-3028] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:47,514 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:55:47,514 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:47,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:47,613 DEBUG [Thread-3028] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:47,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:47,615 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:47,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:47,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:47,616 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:47,616 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:47,616 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:47,616 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:47,616 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:47,617 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:47,621 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:47,621 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:47,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:47,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:47,622 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:47,622 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:47,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:47,622 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:47,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:47,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:47,622 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:47,623 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:55:47,623 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:47,623 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:47,623 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:47,623 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:47,623 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:47,623 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:47,623 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:47,623 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. started... 2023-04-19 20:55:47,623 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:47,623 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. as already flushing 2023-04-19 20:55:47,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35470 deadline: 1681937807613, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:47,623 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:47,624 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:47,624 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:47,624 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:47,624 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:47,624 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:47,626 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:47,626 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:47,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:47,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:47,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:47,627 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:47,627 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:47,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:47,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:47,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:47,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:47,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:47,628 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:47,628 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@590bc285[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:47,628 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:47,628 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:47,629 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:47,629 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:47,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:47,629 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:47,629 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:47,629 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:47,630 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:47,630 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:47,632 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:47,632 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:47,632 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:47,632 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:47,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:47,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:47,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:47,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:47,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:47,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:47,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:47,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:47,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:47,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:47,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:47,634 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:47,634 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:47,634 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:47,634 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:47,634 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:47,635 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:47,645 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:47,645 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:47,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:47,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:47,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:47,645 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:47,645 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:47,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:47,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:47,645 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:47,645 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:47,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:47,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:47,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:47,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:47,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:47,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:47,647 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:47,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:47,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:47,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:47,649 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:47,649 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:47,650 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:47,650 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:47,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:47,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:47,650 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:47,650 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:47,650 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:47,650 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:47,650 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:47,650 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:47,650 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:47,650 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:47,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:47,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:47,650 DEBUG [Thread-3028] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:47,650 DEBUG [Thread-3028] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:47,685 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=506 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/A/1b0d4fd1deee4abab035f01ee9d1d0c0 2023-04-19 20:55:47,750 DEBUG [Thread-3028] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:47,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:47,753 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:47,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:47,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:47,754 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:47,754 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:47,754 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:47,754 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:47,754 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:47,755 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:47,756 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:47,756 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:47,756 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:47,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:47,756 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:47,756 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:47,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:47,757 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:47,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:47,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:47,757 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:47,758 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:55:47,758 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:47,758 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:47,758 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:47,758 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:47,758 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:47,758 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:47,758 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:47,758 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. started... 2023-04-19 20:55:47,758 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. as already flushing 2023-04-19 20:55:47,758 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:47,759 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:47,759 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:47,759 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:47,759 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:47,759 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:47,761 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:47,761 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:47,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:47,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:47,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:47,761 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:47,761 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:47,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:47,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:47,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:47,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:47,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:47,763 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:47,763 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@467b1264[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:47,763 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:47,763 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:47,764 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:47,764 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:47,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:47,764 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:47,764 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:47,764 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:47,764 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:47,764 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:47,767 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:47,767 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:47,767 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:47,767 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:47,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:47,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:47,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:47,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:47,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:47,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:47,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:47,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:47,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:47,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:47,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:47,769 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:47,769 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:47,769 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:47,769 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:47,769 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:47,769 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:47,771 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:47,771 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:47,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:47,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:47,771 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:47,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:47,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:47,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:47,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:47,771 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:47,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:47,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:47,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:47,772 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:47,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:47,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:47,775 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:47,775 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:47,776 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:47,776 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:47,776 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:47,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:47,776 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:47,776 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:47,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:47,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:47,776 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:47,776 DEBUG [Thread-3028] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:47,776 DEBUG [Thread-3028] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:47,776 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:47,776 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:47,776 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:47,776 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:47,777 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:47,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:47,777 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:55:47,833 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:47,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35490 deadline: 1681937807829, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:47,833 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:47,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35474 deadline: 1681937807830, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:47,839 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:47,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35492 deadline: 1681937807835, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:47,876 DEBUG [Thread-3028] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:47,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:47,879 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:47,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:47,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:47,880 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:47,880 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:47,880 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:47,880 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:47,880 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:47,880 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:47,882 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:47,882 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:47,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:47,882 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:47,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:47,882 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:47,882 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:47,883 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:47,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:47,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:47,883 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:47,883 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:55:47,883 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:47,883 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:47,883 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:47,883 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:47,883 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:47,883 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:47,883 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:47,884 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. started... 2023-04-19 20:55:47,884 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. as already flushing 2023-04-19 20:55:47,884 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:47,884 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:47,884 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:47,884 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:47,884 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:47,884 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:47,885 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:47,885 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:47,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:47,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:47,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:47,886 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:47,886 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:47,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:47,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:47,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:47,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:47,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:47,888 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:47,888 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@73432876[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:47,888 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:47,888 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:47,890 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:47,890 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:47,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:47,890 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:47,890 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:47,890 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:47,890 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:47,890 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:47,892 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:47,892 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:47,892 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:47,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:47,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:47,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:47,892 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:47,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:47,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:47,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:47,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:47,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:47,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:47,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:47,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:47,894 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:47,894 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:47,894 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:47,894 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:47,895 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:47,895 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:47,896 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:47,896 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:47,896 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:47,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:47,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:47,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:47,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:47,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:47,897 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:47,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:47,897 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:47,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:47,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:47,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:47,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:47,898 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:47,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:47,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:47,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:47,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:47,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:47,900 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:47,900 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:47,900 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:47,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:47,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:47,900 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:47,900 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:47,900 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:47,900 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:47,900 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:47,900 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:47,900 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:47,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:47,900 DEBUG [Thread-3028] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:47,900 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:55:47,900 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:47,900 DEBUG [Thread-3028] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:47,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:47,900 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:47,929 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:47,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35470 deadline: 1681937807924, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:48,000 DEBUG [Thread-3028] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:48,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:48,002 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:48,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:48,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:48,003 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:48,004 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:48,004 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:48,004 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:48,004 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:48,004 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:48,005 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:48,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:48,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:48,006 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:48,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:48,006 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:48,006 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:48,006 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:48,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:48,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:48,007 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:48,007 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:55:48,007 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:48,007 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:48,007 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:48,007 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:48,007 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:48,007 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:48,007 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:48,007 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. started... 2023-04-19 20:55:48,007 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. as already flushing 2023-04-19 20:55:48,007 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:48,008 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:48,008 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:48,008 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:48,008 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:48,008 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:48,013 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:48,013 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:48,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:48,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:48,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:48,014 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:48,014 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:48,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:48,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:48,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:48,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:48,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:48,016 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:48,016 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ebbdf17[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:48,016 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:48,016 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:48,017 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:48,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:48,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:48,017 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:48,017 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:48,017 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:48,018 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:48,018 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:48,019 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:48,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:48,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:48,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:48,019 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:48,019 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:48,019 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:48,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:48,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:48,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:48,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:48,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:48,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:48,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:48,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:48,022 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:48,022 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:48,022 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:48,022 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:48,022 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:48,022 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:48,024 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:48,024 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:48,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:48,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:48,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:48,024 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:48,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:48,024 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:48,025 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:48,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:48,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:48,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:48,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:48,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:48,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:48,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:48,026 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:48,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:48,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:48,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:48,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:48,029 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:48,029 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:48,029 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:48,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:48,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:48,029 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:48,029 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:48,030 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:48,029 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:48,030 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:48,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:48,030 DEBUG [Thread-3028] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:48,030 DEBUG [Thread-3028] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:48,029 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:48,030 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:48,030 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:48,030 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:48,030 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:48,095 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=506 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/B/f5f5527ed09f4697b20a7cce4ead1353 2023-04-19 20:55:48,130 DEBUG [Thread-3028] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:48,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:48,132 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:48,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:48,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:48,133 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:48,133 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:48,133 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:48,133 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:48,134 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:48,134 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:48,135 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:48,135 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:48,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:48,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:48,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:48,136 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:48,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:48,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:48,136 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:48,137 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:55:48,137 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:48,137 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:48,137 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:48,137 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:48,137 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:48,137 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:48,137 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:48,137 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:48,137 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:48,137 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. started... 2023-04-19 20:55:48,137 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. as already flushing 2023-04-19 20:55:48,137 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:48,137 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:48,137 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:48,137 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:48,137 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:48,137 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:48,139 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:48,139 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:48,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:48,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:48,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:48,139 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:48,139 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:48,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:48,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:48,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:48,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:48,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:48,141 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:48,141 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@792335af[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:48,141 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:48,141 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:48,143 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:48,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:48,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:48,143 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:48,143 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:48,144 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:48,144 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:48,144 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:48,145 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:48,145 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:48,145 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:48,145 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:48,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:48,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:48,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:48,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:48,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:48,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:48,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:48,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:48,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:48,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:48,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:48,148 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:48,148 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:48,148 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:48,148 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:48,148 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:48,148 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:48,150 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:48,150 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:48,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:48,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:48,150 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:48,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:48,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:48,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:48,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:48,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:48,150 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:48,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:48,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:48,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:48,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:48,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:48,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:48,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:48,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:48,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:48,153 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:48,155 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:48,155 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:48,155 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:48,155 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:48,155 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:48,155 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:48,155 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:48,155 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:48,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:48,155 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:48,155 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:48,155 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:48,155 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:48,155 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:48,155 DEBUG [Thread-3028] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:48,155 DEBUG [Thread-3028] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:48,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:48,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:48,256 DEBUG [Thread-3028] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:48,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:48,257 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:48,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:48,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:48,259 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:48,259 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:48,259 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:48,259 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:48,259 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:48,259 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:48,261 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:48,261 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:48,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:48,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:48,261 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:48,261 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:48,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:48,262 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:48,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:48,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:48,262 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:48,262 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:55:48,262 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:48,262 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:48,263 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:48,263 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:48,263 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:48,263 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:48,263 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:48,263 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. started... 2023-04-19 20:55:48,263 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. as already flushing 2023-04-19 20:55:48,263 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:48,263 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:48,263 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:48,263 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:48,263 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:48,263 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:48,265 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:48,265 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:48,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:48,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:48,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:48,265 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:48,265 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:48,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:48,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:48,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:48,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:48,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:48,266 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:48,266 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6755e0dd[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:48,266 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:48,266 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:48,269 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:48,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:48,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:48,269 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:48,269 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:48,269 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:48,269 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:48,269 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:48,270 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:48,270 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:48,270 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:48,270 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:48,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:48,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:48,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:48,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:48,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:48,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:48,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:48,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:48,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:48,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:48,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:48,273 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:48,273 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:48,273 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:48,273 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:48,273 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:48,273 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:48,275 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:48,275 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:48,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:48,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:48,275 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:48,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:48,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:48,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:48,276 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:48,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:48,276 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:48,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:48,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:48,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:48,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:48,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:48,277 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:48,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:48,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:48,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:48,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:48,279 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:48,279 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:48,279 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:48,280 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:48,280 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:48,280 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:48,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:48,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:48,280 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:48,280 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:48,280 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:48,280 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:48,280 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:48,280 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:48,280 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:55:48,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:48,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:48,280 DEBUG [Thread-3028] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:48,280 DEBUG [Thread-3028] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:48,380 DEBUG [Thread-3028] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:48,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:48,383 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:48,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:48,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:48,384 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:48,384 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:48,384 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:48,384 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:48,384 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:48,384 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:48,386 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:48,386 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:48,386 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:48,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:48,387 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:48,387 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:48,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:48,387 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:48,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:48,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:48,387 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:48,388 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:55:48,388 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:48,388 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:48,388 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:48,388 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:48,388 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:48,388 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:48,388 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:48,388 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. started... 2023-04-19 20:55:48,388 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. as already flushing 2023-04-19 20:55:48,389 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:48,389 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:48,389 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:48,389 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:48,389 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:48,389 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:48,390 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:48,390 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:48,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:48,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:48,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:48,390 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:48,390 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:48,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:48,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:48,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:48,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:48,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:48,392 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:48,392 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@d3f64d9[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:48,392 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:48,392 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:48,396 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:48,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:48,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:48,396 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:48,396 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:48,396 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:48,396 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:48,396 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:48,400 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:48,400 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:48,400 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:48,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:48,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:48,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:48,400 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:48,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:48,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:48,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:48,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:48,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:48,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:48,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:48,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:48,403 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:48,403 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:48,403 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:48,403 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:48,403 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:48,403 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:48,404 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:48,404 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:48,404 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:48,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:48,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:48,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:48,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:48,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:48,405 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:48,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:48,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:48,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:48,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:48,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:48,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:48,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:48,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:48,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:48,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:48,408 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:48,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:48,411 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:48,411 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:48,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:48,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:48,411 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:48,411 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:48,411 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:48,411 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:48,411 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:48,411 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:48,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:48,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:48,411 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:48,411 DEBUG [Thread-3028] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:48,411 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:48,412 DEBUG [Thread-3028] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:48,412 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:48,412 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:48,438 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:48,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35470 deadline: 1681937808432, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:48,504 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=506 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/C/6bf6cc3d591d4b06ab6918418e3acdc6 2023-04-19 20:55:48,508 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/A/1b0d4fd1deee4abab035f01ee9d1d0c0 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/1b0d4fd1deee4abab035f01ee9d1d0c0 2023-04-19 20:55:48,512 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/1b0d4fd1deee4abab035f01ee9d1d0c0, entries=200, sequenceid=506, filesize=14.3 K 2023-04-19 20:55:48,512 DEBUG [Thread-3028] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:48,512 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/B/f5f5527ed09f4697b20a7cce4ead1353 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/f5f5527ed09f4697b20a7cce4ead1353 2023-04-19 20:55:48,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:48,515 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:48,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:48,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:48,516 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:48,516 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:48,516 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:48,516 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:48,516 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:48,516 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:48,516 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/f5f5527ed09f4697b20a7cce4ead1353, entries=150, sequenceid=506, filesize=11.9 K 2023-04-19 20:55:48,517 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/C/6bf6cc3d591d4b06ab6918418e3acdc6 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/6bf6cc3d591d4b06ab6918418e3acdc6 2023-04-19 20:55:48,518 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:48,518 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:48,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:48,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:48,518 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:48,518 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:48,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:48,519 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:48,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:48,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:48,519 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:48,519 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:55:48,519 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:48,519 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:48,519 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:48,519 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:48,520 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:48,520 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:48,520 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:48,520 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. started... 2023-04-19 20:55:48,520 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. as already flushing 2023-04-19 20:55:48,520 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:48,520 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:48,520 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:48,520 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:48,520 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:48,520 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:48,520 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/6bf6cc3d591d4b06ab6918418e3acdc6, entries=150, sequenceid=506, filesize=11.9 K 2023-04-19 20:55:48,521 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~419.11 KB/429168, currentSize=33.54 KB/34350 for 4393734ae917e967cf2b5b4b3b44ed0d in 1248ms, sequenceid=506, compaction requested=true 2023-04-19 20:55:48,521 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 4393734ae917e967cf2b5b4b3b44ed0d: 2023-04-19 20:55:48,521 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 20:55:48,521 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-19 20:55:48,521 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-19 20:55:48,521 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 20:55:48,522 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 20:55:48,522 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39612 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-19 20:55:48,522 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1912): 4393734ae917e967cf2b5b4b3b44ed0d/A is initiating minor compaction (all files) 2023-04-19 20:55:48,522 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 4393734ae917e967cf2b5b4b3b44ed0d/A in TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:48,522 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/476f2dffd0a0400d8b8e9e7891445841, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/3d713cd6cad44bf8909494a274ee3256, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/1b0d4fd1deee4abab035f01ee9d1d0c0] into tmpdir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp, totalSize=38.7 K 2023-04-19 20:55:48,523 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37172 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-19 20:55:48,523 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1912): 4393734ae917e967cf2b5b4b3b44ed0d/B is initiating minor compaction (all files) 2023-04-19 20:55:48,523 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 476f2dffd0a0400d8b8e9e7891445841, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=422, earliestPutTs=1681937745986 2023-04-19 20:55:48,523 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4393734ae917e967cf2b5b4b3b44ed0d/B in TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:48,523 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/b25198594675479583e0b463c8587b1c, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/9a2493853ea342f7b2bff95de1f09376, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/f5f5527ed09f4697b20a7cce4ead1353] into tmpdir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp, totalSize=36.3 K 2023-04-19 20:55:48,523 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:48,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:48,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:48,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:48,523 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:48,523 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 3d713cd6cad44bf8909494a274ee3256, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=431, earliestPutTs=1681937746263 2023-04-19 20:55:48,524 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting b25198594675479583e0b463c8587b1c, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=422, earliestPutTs=1681937745986 2023-04-19 20:55:48,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:48,524 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:48,524 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:48,524 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 1b0d4fd1deee4abab035f01ee9d1d0c0, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=506, earliestPutTs=1681937746681 2023-04-19 20:55:48,524 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 9a2493853ea342f7b2bff95de1f09376, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=431, earliestPutTs=1681937746263 2023-04-19 20:55:48,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:48,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:48,525 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting f5f5527ed09f4697b20a7cce4ead1353, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=506, earliestPutTs=1681937746681 2023-04-19 20:55:48,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:48,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:48,525 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:48,525 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2d5c510e[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:48,525 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:48,526 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:48,527 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:48,528 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:48,528 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:48,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:48,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:48,528 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:48,528 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:48,528 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:48,529 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:48,529 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:48,529 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:48,530 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:48,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:48,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:48,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:48,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:48,531 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] throttle.PressureAwareThroughputController(145): 4393734ae917e967cf2b5b4b3b44ed0d#A#compaction#446 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 20:55:48,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:48,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:48,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:48,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:48,533 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] throttle.PressureAwareThroughputController(145): 4393734ae917e967cf2b5b4b3b44ed0d#B#compaction#447 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 20:55:48,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:48,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:48,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:48,534 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:48,534 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:48,534 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:48,534 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:48,534 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:48,534 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:48,536 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:48,536 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:48,536 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:48,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:48,536 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:48,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:48,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:48,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:48,537 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:48,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:48,537 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:48,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:48,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:48,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:48,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:48,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:48,540 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:48,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:48,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:48,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:48,543 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/A/f3aed609eeee412dbad9a1322a48bb8f as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/f3aed609eeee412dbad9a1322a48bb8f 2023-04-19 20:55:48,545 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:48,545 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:48,545 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:48,545 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/B/19dd7570fe064ae1aa15f42c83abebe2 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/19dd7570fe064ae1aa15f42c83abebe2 2023-04-19 20:55:48,545 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:48,545 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:48,545 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:48,545 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:48,545 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:48,545 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:48,545 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:48,545 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:48,545 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:55:48,545 DEBUG [Thread-3028] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:48,546 DEBUG [Thread-3028] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:48,545 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:48,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:48,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:48,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:48,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:48,550 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 4393734ae917e967cf2b5b4b3b44ed0d/A of 4393734ae917e967cf2b5b4b3b44ed0d into f3aed609eeee412dbad9a1322a48bb8f(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 20:55:48,550 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4393734ae917e967cf2b5b4b3b44ed0d: 2023-04-19 20:55:48,550 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d., storeName=4393734ae917e967cf2b5b4b3b44ed0d/A, priority=13, startTime=1681937748521; duration=0sec 2023-04-19 20:55:48,550 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 20:55:48,550 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-19 20:55:48,550 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 4393734ae917e967cf2b5b4b3b44ed0d/B of 4393734ae917e967cf2b5b4b3b44ed0d into 19dd7570fe064ae1aa15f42c83abebe2(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 20:55:48,550 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4393734ae917e967cf2b5b4b3b44ed0d: 2023-04-19 20:55:48,550 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d., storeName=4393734ae917e967cf2b5b4b3b44ed0d/B, priority=13, startTime=1681937748521; duration=0sec 2023-04-19 20:55:48,551 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 20:55:48,551 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37172 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-19 20:55:48,551 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1912): 4393734ae917e967cf2b5b4b3b44ed0d/C is initiating minor compaction (all files) 2023-04-19 20:55:48,551 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 4393734ae917e967cf2b5b4b3b44ed0d/C in TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:48,551 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/717cf7885aff481499668309cb0affbc, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/8aab4a235bb34537849c7f92acaa9ac0, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/6bf6cc3d591d4b06ab6918418e3acdc6] into tmpdir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp, totalSize=36.3 K 2023-04-19 20:55:48,552 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 717cf7885aff481499668309cb0affbc, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=422, earliestPutTs=1681937745986 2023-04-19 20:55:48,552 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 8aab4a235bb34537849c7f92acaa9ac0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=431, earliestPutTs=1681937746263 2023-04-19 20:55:48,552 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 6bf6cc3d591d4b06ab6918418e3acdc6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=506, earliestPutTs=1681937746681 2023-04-19 20:55:48,557 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] throttle.PressureAwareThroughputController(145): 4393734ae917e967cf2b5b4b3b44ed0d#C#compaction#448 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 20:55:48,569 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/C/537489db7e744a4b89644c6fb462bc3c as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/537489db7e744a4b89644c6fb462bc3c 2023-04-19 20:55:48,575 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 4393734ae917e967cf2b5b4b3b44ed0d/C of 4393734ae917e967cf2b5b4b3b44ed0d into 537489db7e744a4b89644c6fb462bc3c(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 20:55:48,575 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4393734ae917e967cf2b5b4b3b44ed0d: 2023-04-19 20:55:48,575 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d., storeName=4393734ae917e967cf2b5b4b3b44ed0d/C, priority=13, startTime=1681937748522; duration=0sec 2023-04-19 20:55:48,575 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 20:55:48,646 DEBUG [Thread-3028] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:48,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:48,648 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:48,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:48,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:48,649 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:48,649 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:48,649 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:48,649 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:48,649 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:48,649 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:48,651 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:48,651 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:48,651 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:48,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:48,651 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:48,651 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:48,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:48,652 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:48,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:48,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:48,652 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:48,652 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:55:48,652 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:48,652 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:48,653 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:48,653 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:48,653 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:48,653 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:48,653 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:48,653 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. started... 2023-04-19 20:55:48,653 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4393734ae917e967cf2b5b4b3b44ed0d 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-19 20:55:48,653 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4393734ae917e967cf2b5b4b3b44ed0d, store=A 2023-04-19 20:55:48,653 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:55:48,653 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4393734ae917e967cf2b5b4b3b44ed0d, store=B 2023-04-19 20:55:48,653 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:55:48,653 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4393734ae917e967cf2b5b4b3b44ed0d, store=C 2023-04-19 20:55:48,653 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:55:48,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(9158): Flush requested on 4393734ae917e967cf2b5b4b3b44ed0d 2023-04-19 20:55:48,872 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. as already flushing 2023-04-19 20:55:48,905 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681937748893, max timestamp=1681937748904Num uniques 859; 2023-04-19 20:55:48,905 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 20:55:48,906 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681937748893, max timestamp=1681937748904Num uniques 859; 2023-04-19 20:55:48,906 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 20:55:48,906 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681937748893, max timestamp=1681937748904Num uniques 859; 2023-04-19 20:55:48,906 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 20:55:48,943 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1681937748928, max timestamp=1681937748940Num uniques 1018; 2023-04-19 20:55:48,943 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1681937748928, max timestamp=1681937748940Num uniques 1018; 2023-04-19 20:55:48,943 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=12268, suffix heap size=133840,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=268 2023-04-19 20:55:48,943 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=12268, suffix heap size=133840,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=268 2023-04-19 20:55:48,943 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1681937748928, max timestamp=1681937748940Num uniques 1018; 2023-04-19 20:55:48,943 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=12268, suffix heap size=133840,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=268 2023-04-19 20:55:48,980 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1681937748960, max timestamp=1681937748977Num uniques 1127; 2023-04-19 20:55:48,980 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1681937748960, max timestamp=1681937748977Num uniques 1127; 2023-04-19 20:55:48,980 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1681937748960, max timestamp=1681937748977Num uniques 1127; 2023-04-19 20:55:48,980 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=10392, suffix heap size=145176,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=227 2023-04-19 20:55:48,980 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=10392, suffix heap size=145176,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=227 2023-04-19 20:55:48,980 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=10392, suffix heap size=145176,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=227 2023-04-19 20:55:49,017 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:49,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35474 deadline: 1681937809011, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:49,019 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:49,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35492 deadline: 1681937809012, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:49,019 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1681937749005, max timestamp=1681937749016Num uniques 1086; 2023-04-19 20:55:49,019 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1681937749005, max timestamp=1681937749016Num uniques 1086; 2023-04-19 20:55:49,019 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=8516, suffix heap size=140912,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=186 2023-04-19 20:55:49,019 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1681937749005, max timestamp=1681937749016Num uniques 1086; 2023-04-19 20:55:49,019 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=8516, suffix heap size=140912,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=186 2023-04-19 20:55:49,019 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=8516, suffix heap size=140912,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=186 2023-04-19 20:55:49,062 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=517 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/A/ae821819bef4404294bc870a1597cf64 2023-04-19 20:55:49,124 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1681937749091, max timestamp=1681937749121Num uniques 1045; 2023-04-19 20:55:49,124 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1681937749091, max timestamp=1681937749121Num uniques 1045; 2023-04-19 20:55:49,124 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1681937749091, max timestamp=1681937749121Num uniques 1045; 2023-04-19 20:55:49,124 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-04-19 20:55:49,124 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-04-19 20:55:49,124 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-04-19 20:55:49,170 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=254, cellSize=11630, totalHeapSize=26656, min timestamp=1681937749154, max timestamp=1681937749167Num uniques 1054; 2023-04-19 20:55:49,170 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=254, cellSize=11630, totalHeapSize=26656, min timestamp=1681937749154, max timestamp=1681937749167Num uniques 1054; 2023-04-19 20:55:49,170 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=254, cellSize=11630, totalHeapSize=26656, min timestamp=1681937749154, max timestamp=1681937749167Num uniques 1054; 2023-04-19 20:55:49,170 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=11630, suffix heap size=137584,new segment heap size=26656 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=254 2023-04-19 20:55:49,170 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=11630, suffix heap size=137584,new segment heap size=26656 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=254 2023-04-19 20:55:49,170 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=11630, suffix heap size=137584,new segment heap size=26656 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=254 2023-04-19 20:55:49,224 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:49,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35492 deadline: 1681937809213, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:49,224 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:49,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35490 deadline: 1681937809214, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:49,224 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:49,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35474 deadline: 1681937809214, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:49,330 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:49,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35490 deadline: 1681937809325, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:49,330 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:49,330 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:49,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35474 deadline: 1681937809325, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:49,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35492 deadline: 1681937809325, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:49,445 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:49,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35470 deadline: 1681937809439, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:49,472 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=517 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/B/45f2ee1ae15e413088ef7d40130256b8 2023-04-19 20:55:49,536 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:49,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35490 deadline: 1681937809531, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:49,537 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:49,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35492 deadline: 1681937809532, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:49,537 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:49,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35474 deadline: 1681937809532, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:49,844 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:49,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35492 deadline: 1681937809838, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:49,844 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:49,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35490 deadline: 1681937809838, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:49,845 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:49,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35474 deadline: 1681937809839, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:49,881 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=517 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/C/9d6ecbee77b84887bde1852370de6152 2023-04-19 20:55:49,885 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/A/ae821819bef4404294bc870a1597cf64 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/ae821819bef4404294bc870a1597cf64 2023-04-19 20:55:49,889 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/ae821819bef4404294bc870a1597cf64, entries=150, sequenceid=517, filesize=11.9 K 2023-04-19 20:55:49,890 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/B/45f2ee1ae15e413088ef7d40130256b8 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/45f2ee1ae15e413088ef7d40130256b8 2023-04-19 20:55:49,893 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/45f2ee1ae15e413088ef7d40130256b8, entries=150, sequenceid=517, filesize=11.9 K 2023-04-19 20:55:49,893 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/C/9d6ecbee77b84887bde1852370de6152 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/9d6ecbee77b84887bde1852370de6152 2023-04-19 20:55:49,896 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/9d6ecbee77b84887bde1852370de6152, entries=150, sequenceid=517, filesize=11.9 K 2023-04-19 20:55:49,897 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=147.60 KB/151140 for 4393734ae917e967cf2b5b4b3b44ed0d in 1244ms, sequenceid=517, compaction requested=false 2023-04-19 20:55:49,897 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4393734ae917e967cf2b5b4b3b44ed0d: 2023-04-19 20:55:49,897 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:49,897 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:49,897 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:49,897 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:49,897 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:49,897 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:49,900 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:49,900 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:49,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:49,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:49,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:49,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:49,900 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:49,901 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:49,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:49,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:49,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:49,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:49,902 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:49,902 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4db28747[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:49,902 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:49,902 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:49,903 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:49,903 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:49,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:49,903 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:49,903 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:49,903 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:49,903 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:49,903 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:49,905 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:49,905 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:49,905 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:49,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:49,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:49,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:49,905 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:49,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:49,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:49,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:49,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:49,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:49,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:49,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:49,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:49,907 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:49,907 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:49,907 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:49,907 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:49,907 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:49,907 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:49,909 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:49,909 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:49,909 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:49,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:49,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:49,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:49,909 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:49,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:49,910 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:49,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:49,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:49,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:49,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:49,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:49,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:49,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:49,911 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:49,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:49,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:49,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:49,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:49,914 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:49,914 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:49,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:49,914 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:49,914 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:49,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:49,914 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:49,914 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:49,914 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:49,914 DEBUG [Thread-3028] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:49,915 DEBUG [Thread-3028] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:49,914 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:49,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:49,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:49,914 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:49,915 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:49,915 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:49,915 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:50,015 DEBUG [Thread-3028] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:50,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:50,017 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:50,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:50,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:50,018 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:50,018 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:50,018 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:50,018 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:50,019 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:50,019 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:50,020 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:50,020 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:50,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:50,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:50,020 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:50,020 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:50,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:50,020 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:50,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:50,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:50,021 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:50,021 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:55:50,021 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:50,021 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:50,021 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:50,021 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:50,021 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:50,021 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:50,021 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:50,021 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. started... 2023-04-19 20:55:50,021 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4393734ae917e967cf2b5b4b3b44ed0d 3/3 column families, dataSize=147.60 KB heapSize=415.92 KB 2023-04-19 20:55:50,022 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4393734ae917e967cf2b5b4b3b44ed0d, store=A 2023-04-19 20:55:50,022 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 20:55:50,022 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4393734ae917e967cf2b5b4b3b44ed0d, store=B 2023-04-19 20:55:50,022 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 20:55:50,022 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4393734ae917e967cf2b5b4b3b44ed0d, store=C 2023-04-19 20:55:50,022 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 20:55:50,030 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=640 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/A/5233ec518803433bad5c9d0efa7ea40f 2023-04-19 20:55:50,040 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=640 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/B/766ca41fef7a4f9fa07fd0f5deac3107 2023-04-19 20:55:50,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(9158): Flush requested on 4393734ae917e967cf2b5b4b3b44ed0d 2023-04-19 20:55:50,350 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. as already flushing 2023-04-19 20:55:50,358 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:50,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35492 deadline: 1681937810352, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:50,358 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:50,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35474 deadline: 1681937810353, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:50,360 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:50,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35490 deadline: 1681937810358, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:50,450 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=640 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/C/bc6301135b994e2c9b159011972897b8 2023-04-19 20:55:50,454 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/A/5233ec518803433bad5c9d0efa7ea40f as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/5233ec518803433bad5c9d0efa7ea40f 2023-04-19 20:55:50,457 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/5233ec518803433bad5c9d0efa7ea40f, entries=150, sequenceid=640, filesize=11.9 K 2023-04-19 20:55:50,458 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/B/766ca41fef7a4f9fa07fd0f5deac3107 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/766ca41fef7a4f9fa07fd0f5deac3107 2023-04-19 20:55:50,461 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/766ca41fef7a4f9fa07fd0f5deac3107, entries=150, sequenceid=640, filesize=11.9 K 2023-04-19 20:55:50,462 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/C/bc6301135b994e2c9b159011972897b8 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/bc6301135b994e2c9b159011972897b8 2023-04-19 20:55:50,465 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/bc6301135b994e2c9b159011972897b8, entries=150, sequenceid=640, filesize=11.9 K 2023-04-19 20:55:50,465 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~415.88 KB/425856, currentSize=33.54 KB/34350 for 4393734ae917e967cf2b5b4b3b44ed0d in 444ms, sequenceid=640, compaction requested=true 2023-04-19 20:55:50,465 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4393734ae917e967cf2b5b4b3b44ed0d: 2023-04-19 20:55:50,465 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:50,465 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:50,466 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:50,466 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:50,466 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:50,466 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:50,467 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:50,467 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:50,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:50,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:50,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:50,468 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:50,468 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:50,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:50,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:50,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(9158): Flush requested on 4393734ae917e967cf2b5b4b3b44ed0d 2023-04-19 20:55:50,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:50,469 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 4393734ae917e967cf2b5b4b3b44ed0d 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-19 20:55:50,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:50,469 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4393734ae917e967cf2b5b4b3b44ed0d, store=A 2023-04-19 20:55:50,470 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:55:50,470 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4393734ae917e967cf2b5b4b3b44ed0d, store=B 2023-04-19 20:55:50,470 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:55:50,470 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4393734ae917e967cf2b5b4b3b44ed0d, store=C 2023-04-19 20:55:50,470 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:55:50,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:50,470 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:50,470 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5a9c524[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:50,470 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:50,471 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:50,472 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:50,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:50,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:50,472 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:50,472 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:50,472 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:50,473 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:50,473 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:50,476 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:50,476 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:50,476 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:50,476 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:50,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:50,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:50,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:50,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:50,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:50,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:50,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:50,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:50,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:50,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:50,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:50,478 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:50,478 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:50,479 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:50,479 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:50,479 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:50,479 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:50,480 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=651 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/A/59bac3d9b1b34d1188aba5c0152c0f59 2023-04-19 20:55:50,482 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:50,482 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:50,482 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:50,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:50,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:50,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:50,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:50,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:50,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:50,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:50,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:50,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:50,484 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:50,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:50,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:50,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:50,485 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:50,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:50,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:50,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:50,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:50,487 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:50,487 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:50,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:50,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:50,487 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:50,487 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:50,487 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:50,487 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:50,487 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:50,487 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:50,487 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:50,488 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:50,488 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:55:50,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:50,488 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:50,488 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:50,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:50,488 DEBUG [Thread-3028] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:50,488 DEBUG [Thread-3028] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:50,525 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681937750497, max timestamp=1681937750524Num uniques 859; 2023-04-19 20:55:50,526 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-19 20:55:50,526 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681937750497, max timestamp=1681937750524Num uniques 859; 2023-04-19 20:55:50,526 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-19 20:55:50,526 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681937750497, max timestamp=1681937750524Num uniques 859; 2023-04-19 20:55:50,526 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-19 20:55:50,567 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:50,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35490 deadline: 1681937810556, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:50,567 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:50,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35492 deadline: 1681937810560, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:50,567 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:50,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 245 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35474 deadline: 1681937810564, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:50,588 DEBUG [Thread-3028] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:50,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:50,590 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:50,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:50,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:50,591 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:50,591 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:50,591 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:50,591 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:50,592 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:50,592 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:50,594 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:50,594 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:50,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:50,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:50,594 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:50,594 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:50,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:50,595 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:50,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:50,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:50,595 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:50,595 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:55:50,595 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:50,595 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:50,596 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:50,596 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:50,596 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:50,596 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:50,596 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:50,596 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. started... 2023-04-19 20:55:50,596 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. as already flushing 2023-04-19 20:55:50,596 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:50,596 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:50,596 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:50,596 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:50,596 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:50,596 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:50,599 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:50,599 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:50,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:50,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:50,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:50,599 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:50,599 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:50,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:50,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:50,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:50,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:50,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:50,601 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:50,601 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@73c660e4[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:50,601 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:50,601 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:50,602 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:50,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:50,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:50,602 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:50,602 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:50,602 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:50,602 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:50,602 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:50,603 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:50,603 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:50,603 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:50,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:50,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:50,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:50,603 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:50,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:50,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:50,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:50,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:50,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:50,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:50,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:50,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:50,606 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:50,606 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:50,606 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:50,606 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:50,606 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:50,606 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:50,609 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:50,609 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:50,609 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:50,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:50,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:50,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:50,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:50,610 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:50,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:50,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:50,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:50,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:50,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:50,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:50,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:50,611 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:50,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:50,613 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:50,613 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:50,613 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:50,613 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:50,613 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:50,613 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:50,613 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:50,613 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:50,613 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:50,613 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:50,613 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:50,613 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:55:50,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:50,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:50,613 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:50,613 DEBUG [Thread-3028] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:50,613 DEBUG [Thread-3028] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:50,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:50,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:50,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:50,614 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:55:50,674 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:50,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 241 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35490 deadline: 1681937810668, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:50,674 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:50,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35492 deadline: 1681937810669, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:50,674 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:50,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 247 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35474 deadline: 1681937810669, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:50,713 DEBUG [Thread-3028] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:50,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:50,715 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:50,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:50,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:50,720 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:50,720 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:50,720 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:50,720 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:50,720 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:50,720 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:50,722 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:50,722 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:50,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:50,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:50,722 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:50,722 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:50,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:50,724 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:50,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:50,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:50,724 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:50,725 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:55:50,725 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:50,725 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:50,725 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:50,725 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:50,725 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:50,725 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:50,725 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:50,725 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. started... 2023-04-19 20:55:50,725 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. as already flushing 2023-04-19 20:55:50,725 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:50,725 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:50,725 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:50,725 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:50,726 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:50,726 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:50,727 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:50,727 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:50,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:50,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:50,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:50,728 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:50,728 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:50,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:50,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:50,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:50,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:50,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:50,729 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:50,729 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3c5d8bb3[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:50,729 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:50,729 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:50,731 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:50,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:50,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:50,732 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:50,732 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:50,732 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:50,732 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:50,732 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:50,733 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:50,733 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:50,733 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:50,733 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:50,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:50,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:50,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:50,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:50,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:50,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:50,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:50,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:50,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:50,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:50,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:50,737 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:50,737 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:50,737 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:50,737 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:50,737 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:50,737 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:50,738 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:50,738 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:50,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:50,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:50,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:50,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:50,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:50,738 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:50,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:50,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:50,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:50,739 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:50,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:50,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:50,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:50,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:50,741 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:50,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:50,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:50,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:50,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:50,743 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:50,743 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:50,743 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:50,743 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:50,743 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:50,743 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:50,743 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:50,743 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:50,743 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:50,743 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:50,743 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:50,743 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:55:50,743 DEBUG [Thread-3028] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:50,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:50,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:50,743 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:50,743 DEBUG [Thread-3028] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:50,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:50,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:50,844 DEBUG [Thread-3028] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:50,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:50,846 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:50,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:50,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:50,847 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:50,847 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:50,847 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:50,847 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:50,847 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:50,847 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:50,849 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:50,849 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:50,849 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:50,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:50,849 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:50,849 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:50,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:50,850 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:50,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:50,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:50,850 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:50,850 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:55:50,850 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:50,850 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:50,850 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:50,851 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:50,851 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:50,851 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:50,851 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:50,851 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. started... 2023-04-19 20:55:50,851 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. as already flushing 2023-04-19 20:55:50,851 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:50,851 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:50,851 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:50,851 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:50,851 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:50,851 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:50,853 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:50,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:50,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:50,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:50,853 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:50,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:50,854 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:50,854 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:50,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:50,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:50,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:50,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:50,855 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:50,855 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@658cabc0[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:50,855 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:50,855 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:50,856 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:50,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:50,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:50,856 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:50,856 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:50,856 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:50,857 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:50,857 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:50,858 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:50,858 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:50,858 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:50,858 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:50,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:50,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:50,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:50,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:50,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:50,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:50,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:50,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:50,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:50,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:50,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:50,860 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:50,860 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:50,860 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:50,860 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:50,860 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:50,861 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:50,863 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:50,863 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:50,863 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:50,863 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:50,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:50,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:50,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:50,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:50,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:50,864 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:50,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:50,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:50,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:50,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:50,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:50,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:50,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:50,865 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:50,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:50,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:50,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:50,869 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:50,869 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:50,869 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:50,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:50,869 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:50,869 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:50,869 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:55:50,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:50,869 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:50,869 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:50,870 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:50,870 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:50,870 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:50,870 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:50,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:50,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:50,870 DEBUG [Thread-3028] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:50,870 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:50,870 DEBUG [Thread-3028] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:50,878 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:50,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 243 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35490 deadline: 1681937810876, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:50,882 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:50,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35492 deadline: 1681937810876, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:50,882 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:50,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 249 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35474 deadline: 1681937810876, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:50,897 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=651 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/B/edde5ed5b4a04bebb9a4de18010ec55b 2023-04-19 20:55:50,905 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=651 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/C/fefd7d901c534a8a8ac20a1ac0bbbf70 2023-04-19 20:55:50,909 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/A/59bac3d9b1b34d1188aba5c0152c0f59 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/59bac3d9b1b34d1188aba5c0152c0f59 2023-04-19 20:55:50,912 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/59bac3d9b1b34d1188aba5c0152c0f59, entries=150, sequenceid=651, filesize=11.9 K 2023-04-19 20:55:50,912 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/B/edde5ed5b4a04bebb9a4de18010ec55b as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/edde5ed5b4a04bebb9a4de18010ec55b 2023-04-19 20:55:50,917 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/edde5ed5b4a04bebb9a4de18010ec55b, entries=100, sequenceid=651, filesize=9.5 K 2023-04-19 20:55:50,918 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/C/fefd7d901c534a8a8ac20a1ac0bbbf70 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/fefd7d901c534a8a8ac20a1ac0bbbf70 2023-04-19 20:55:50,921 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/fefd7d901c534a8a8ac20a1ac0bbbf70, entries=100, sequenceid=651, filesize=9.5 K 2023-04-19 20:55:50,921 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 4393734ae917e967cf2b5b4b3b44ed0d in 452ms, sequenceid=651, compaction requested=true 2023-04-19 20:55:50,921 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 4393734ae917e967cf2b5b4b3b44ed0d: 2023-04-19 20:55:50,922 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 20:55:50,922 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 20:55:50,922 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 20:55:50,922 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-19 20:55:50,922 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-19 20:55:50,923 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49456 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-19 20:55:50,923 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1912): 4393734ae917e967cf2b5b4b3b44ed0d/A is initiating minor compaction (all files) 2023-04-19 20:55:50,923 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 4393734ae917e967cf2b5b4b3b44ed0d/A in TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:50,923 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/f3aed609eeee412dbad9a1322a48bb8f, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/ae821819bef4404294bc870a1597cf64, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/5233ec518803433bad5c9d0efa7ea40f, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/59bac3d9b1b34d1188aba5c0152c0f59] into tmpdir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp, totalSize=48.3 K 2023-04-19 20:55:50,923 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47016 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-19 20:55:50,923 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1912): 4393734ae917e967cf2b5b4b3b44ed0d/B is initiating minor compaction (all files) 2023-04-19 20:55:50,923 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4393734ae917e967cf2b5b4b3b44ed0d/B in TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:50,924 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/19dd7570fe064ae1aa15f42c83abebe2, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/45f2ee1ae15e413088ef7d40130256b8, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/766ca41fef7a4f9fa07fd0f5deac3107, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/edde5ed5b4a04bebb9a4de18010ec55b] into tmpdir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp, totalSize=45.9 K 2023-04-19 20:55:50,924 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting f3aed609eeee412dbad9a1322a48bb8f, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=506, earliestPutTs=1681937746681 2023-04-19 20:55:50,924 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 19dd7570fe064ae1aa15f42c83abebe2, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=506, earliestPutTs=1681937746681 2023-04-19 20:55:50,924 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting ae821819bef4404294bc870a1597cf64, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=517, earliestPutTs=1681937747287 2023-04-19 20:55:50,925 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 45f2ee1ae15e413088ef7d40130256b8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=517, earliestPutTs=1681937747287 2023-04-19 20:55:50,925 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 5233ec518803433bad5c9d0efa7ea40f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=640, earliestPutTs=1681937749190 2023-04-19 20:55:50,925 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 766ca41fef7a4f9fa07fd0f5deac3107, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=640, earliestPutTs=1681937749190 2023-04-19 20:55:50,925 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 59bac3d9b1b34d1188aba5c0152c0f59, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=651, earliestPutTs=1681937750466 2023-04-19 20:55:50,925 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting edde5ed5b4a04bebb9a4de18010ec55b, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=651, earliestPutTs=1681937750467 2023-04-19 20:55:50,932 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] throttle.PressureAwareThroughputController(145): 4393734ae917e967cf2b5b4b3b44ed0d#B#compaction#459 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-19 20:55:50,932 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] throttle.PressureAwareThroughputController(145): 4393734ae917e967cf2b5b4b3b44ed0d#A#compaction#458 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 20:55:50,944 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/A/2eb1ff62a6064687a8818a54b4ce106b as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/2eb1ff62a6064687a8818a54b4ce106b 2023-04-19 20:55:50,949 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 4393734ae917e967cf2b5b4b3b44ed0d/A of 4393734ae917e967cf2b5b4b3b44ed0d into 2eb1ff62a6064687a8818a54b4ce106b(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 20:55:50,949 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4393734ae917e967cf2b5b4b3b44ed0d: 2023-04-19 20:55:50,949 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d., storeName=4393734ae917e967cf2b5b4b3b44ed0d/A, priority=12, startTime=1681937750922; duration=0sec 2023-04-19 20:55:50,949 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 20:55:50,949 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-19 20:55:50,950 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47016 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-19 20:55:50,950 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1912): 4393734ae917e967cf2b5b4b3b44ed0d/C is initiating minor compaction (all files) 2023-04-19 20:55:50,950 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 4393734ae917e967cf2b5b4b3b44ed0d/C in TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:50,950 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/537489db7e744a4b89644c6fb462bc3c, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/9d6ecbee77b84887bde1852370de6152, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/bc6301135b994e2c9b159011972897b8, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/fefd7d901c534a8a8ac20a1ac0bbbf70] into tmpdir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp, totalSize=45.9 K 2023-04-19 20:55:50,950 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 537489db7e744a4b89644c6fb462bc3c, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=506, earliestPutTs=1681937746681 2023-04-19 20:55:50,951 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 9d6ecbee77b84887bde1852370de6152, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=517, earliestPutTs=1681937747287 2023-04-19 20:55:50,951 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting bc6301135b994e2c9b159011972897b8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=640, earliestPutTs=1681937749190 2023-04-19 20:55:50,951 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting fefd7d901c534a8a8ac20a1ac0bbbf70, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=651, earliestPutTs=1681937750467 2023-04-19 20:55:50,959 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] throttle.PressureAwareThroughputController(145): 4393734ae917e967cf2b5b4b3b44ed0d#C#compaction#460 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 20:55:50,969 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/C/47fd38401d6e4e24bdc64b779e839c09 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/47fd38401d6e4e24bdc64b779e839c09 2023-04-19 20:55:50,971 DEBUG [Thread-3028] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:50,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:50,973 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:50,973 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 4393734ae917e967cf2b5b4b3b44ed0d/C of 4393734ae917e967cf2b5b4b3b44ed0d into 47fd38401d6e4e24bdc64b779e839c09(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 20:55:50,973 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4393734ae917e967cf2b5b4b3b44ed0d: 2023-04-19 20:55:50,973 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d., storeName=4393734ae917e967cf2b5b4b3b44ed0d/C, priority=12, startTime=1681937750922; duration=0sec 2023-04-19 20:55:50,974 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 20:55:50,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:50,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:50,975 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:50,975 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:50,975 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:50,975 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:50,975 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:50,975 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:50,977 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:50,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:50,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:50,977 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:50,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:50,977 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:50,977 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:50,978 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:50,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:50,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:50,978 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:50,980 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:55:50,980 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:50,980 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:50,980 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:50,980 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:50,980 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:50,980 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:50,980 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:50,980 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. started... 2023-04-19 20:55:50,981 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4393734ae917e967cf2b5b4b3b44ed0d 3/3 column families, dataSize=127.47 KB heapSize=360.38 KB 2023-04-19 20:55:50,981 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4393734ae917e967cf2b5b4b3b44ed0d, store=A 2023-04-19 20:55:50,981 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 20:55:50,981 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4393734ae917e967cf2b5b4b3b44ed0d, store=B 2023-04-19 20:55:50,981 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 20:55:50,981 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4393734ae917e967cf2b5b4b3b44ed0d, store=C 2023-04-19 20:55:50,981 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 20:55:50,994 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=688 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/A/2762c960aec24d86bfb7c0a075a48a78 2023-04-19 20:55:51,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(9158): Flush requested on 4393734ae917e967cf2b5b4b3b44ed0d 2023-04-19 20:55:51,101 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. as already flushing 2023-04-19 20:55:51,166 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:51,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35486 deadline: 1681937811154, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:51,186 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:51,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 245 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35490 deadline: 1681937811181, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:51,189 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:51,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 251 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35474 deadline: 1681937811184, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:51,189 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:51,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35492 deadline: 1681937811185, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:51,272 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:51,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35486 deadline: 1681937811267, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:51,354 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/B/5aef647ab52c412fb92806e34a67b515 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/5aef647ab52c412fb92806e34a67b515 2023-04-19 20:55:51,358 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 4393734ae917e967cf2b5b4b3b44ed0d/B of 4393734ae917e967cf2b5b4b3b44ed0d into 5aef647ab52c412fb92806e34a67b515(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 20:55:51,358 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4393734ae917e967cf2b5b4b3b44ed0d: 2023-04-19 20:55:51,358 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d., storeName=4393734ae917e967cf2b5b4b3b44ed0d/B, priority=12, startTime=1681937750922; duration=0sec 2023-04-19 20:55:51,358 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 20:55:51,414 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=688 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/B/4ac0391327d644b2b7c688dbbce1f03e 2023-04-19 20:55:51,424 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=688 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/C/e823084c3f004185a2ef1ced67cafc9a 2023-04-19 20:55:51,428 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/A/2762c960aec24d86bfb7c0a075a48a78 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/2762c960aec24d86bfb7c0a075a48a78 2023-04-19 20:55:51,431 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/2762c960aec24d86bfb7c0a075a48a78, entries=150, sequenceid=688, filesize=11.9 K 2023-04-19 20:55:51,431 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/B/4ac0391327d644b2b7c688dbbce1f03e as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/4ac0391327d644b2b7c688dbbce1f03e 2023-04-19 20:55:51,434 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/4ac0391327d644b2b7c688dbbce1f03e, entries=150, sequenceid=688, filesize=11.9 K 2023-04-19 20:55:51,435 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/C/e823084c3f004185a2ef1ced67cafc9a as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/e823084c3f004185a2ef1ced67cafc9a 2023-04-19 20:55:51,438 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/e823084c3f004185a2ef1ced67cafc9a, entries=150, sequenceid=688, filesize=11.9 K 2023-04-19 20:55:51,438 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~360.33 KB/368976, currentSize=53.67 KB/54960 for 4393734ae917e967cf2b5b4b3b44ed0d in 458ms, sequenceid=688, compaction requested=false 2023-04-19 20:55:51,439 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4393734ae917e967cf2b5b4b3b44ed0d: 2023-04-19 20:55:51,439 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:51,439 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:51,439 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:51,439 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:51,439 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:51,439 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:51,441 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:51,442 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:51,442 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:51,442 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:51,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:51,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:51,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:51,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:51,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:51,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:51,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:51,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:51,445 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:51,445 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2ec7d2b0[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:51,445 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:51,445 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:51,446 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:51,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:51,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:51,447 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:51,447 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:51,447 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:51,447 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:51,447 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:51,449 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:51,449 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:51,449 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:51,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:51,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:51,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:51,449 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:51,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:51,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:51,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:51,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:51,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:51,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:51,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:51,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:51,452 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:51,452 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:51,453 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:51,453 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:51,453 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:51,453 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:51,454 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:51,454 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:51,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:51,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:51,455 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:51,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:51,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:51,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:51,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:51,455 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:51,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:51,455 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:51,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:51,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:51,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:51,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:51,456 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:51,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:51,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:51,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:51,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:51,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(9158): Flush requested on 4393734ae917e967cf2b5b4b3b44ed0d 2023-04-19 20:55:51,460 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 4393734ae917e967cf2b5b4b3b44ed0d 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-19 20:55:51,460 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4393734ae917e967cf2b5b4b3b44ed0d, store=A 2023-04-19 20:55:51,461 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:55:51,461 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4393734ae917e967cf2b5b4b3b44ed0d, store=B 2023-04-19 20:55:51,461 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:55:51,461 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4393734ae917e967cf2b5b4b3b44ed0d, store=C 2023-04-19 20:55:51,461 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:55:51,462 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:51,462 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:51,462 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:51,462 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:51,462 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:51,462 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:51,462 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:51,462 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:51,462 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:51,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:51,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:51,462 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:51,462 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:51,462 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:51,462 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:55:51,462 DEBUG [Thread-3028] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:51,462 DEBUG [Thread-3028] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:51,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:51,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:51,472 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=701 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/A/15792f21d70c4057826d9060c2ec1bb6 2023-04-19 20:55:51,494 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=701 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/B/9739c501febe47e99417caa65c62a5fb 2023-04-19 20:55:51,531 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:51,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35470 deadline: 1681937811527, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:51,533 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:51,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35486 deadline: 1681937811527, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:51,563 DEBUG [Thread-3028] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:51,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:51,565 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:51,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:51,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:51,566 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:51,566 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:51,566 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:51,566 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:51,566 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:51,566 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:51,568 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:51,568 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:51,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:51,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:51,568 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:51,568 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:51,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:51,569 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:51,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:51,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:51,569 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:51,569 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:55:51,569 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:51,569 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:51,569 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:51,569 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:51,570 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:51,570 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:51,570 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:51,570 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. started... 2023-04-19 20:55:51,570 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. as already flushing 2023-04-19 20:55:51,570 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:51,570 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:51,570 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:51,570 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:51,570 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:51,570 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:51,572 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:51,572 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:51,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:51,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:51,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:51,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:51,573 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:51,573 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:51,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:51,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:51,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:51,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:51,574 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:51,574 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ec656cc[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:51,574 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:51,574 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:51,575 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:51,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:51,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:51,575 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:51,575 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:51,575 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:51,576 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:51,576 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:51,578 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:51,578 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:51,578 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:51,578 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:51,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:51,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:51,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:51,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:51,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:51,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:51,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:51,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:51,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:51,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:51,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:51,581 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:51,581 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:51,581 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:51,581 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:51,581 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:51,581 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:51,583 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:51,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:51,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:51,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:51,583 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:51,583 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:51,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:51,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:51,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:51,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:51,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:51,583 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:51,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:51,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:51,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:51,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:51,584 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:51,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:51,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:51,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:51,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:51,586 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:51,586 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:51,587 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:51,587 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:51,587 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:51,587 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:51,587 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:55:51,587 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:51,587 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:51,587 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:51,587 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:51,587 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:51,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:51,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:51,587 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:51,587 DEBUG [Thread-3028] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:51,587 DEBUG [Thread-3028] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:51,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:51,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:51,638 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:51,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35470 deadline: 1681937811632, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:51,640 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:51,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35486 deadline: 1681937811634, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:51,687 DEBUG [Thread-3028] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:51,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:51,690 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:51,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:51,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:51,691 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:51,691 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:51,691 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:51,691 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:51,691 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:51,691 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:51,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 247 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35490 deadline: 1681937811690, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:51,692 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:51,693 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:51,693 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:51,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:51,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:51,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:51,694 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:51,694 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:51,694 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:51,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:51,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:51,694 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:51,694 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:55:51,694 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:51,695 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:51,695 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:51,695 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:51,695 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:51,695 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:51,695 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:51,695 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. started... 2023-04-19 20:55:51,695 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. as already flushing 2023-04-19 20:55:51,695 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:51,695 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:51,695 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:51,695 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:51,695 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:51,695 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:51,698 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:51,698 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:51,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:51,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:51,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:51,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:51,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:51,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:51,698 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:51,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 253 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35474 deadline: 1681937811694, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:51,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:51,699 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:51,699 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:51,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35492 deadline: 1681937811694, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:51,699 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:51,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:51,700 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:51,700 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3e9658b4[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:51,700 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:51,700 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:51,701 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:51,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:51,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:51,701 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:51,701 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:51,701 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:51,701 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:51,701 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:51,702 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:51,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:51,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:51,702 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:51,703 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:51,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:51,703 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:51,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:51,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:51,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:51,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:51,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:51,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:51,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:51,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:51,705 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:51,705 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:51,705 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:51,705 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:51,705 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:51,705 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:51,707 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:51,707 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:51,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:51,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:51,707 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:51,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:51,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:51,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:51,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:51,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:51,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:51,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:51,708 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:51,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:51,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:51,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:51,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:51,709 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:51,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:51,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:51,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:51,714 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:51,714 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:51,714 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:51,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:51,714 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:51,715 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:51,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:51,714 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:51,714 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:51,715 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:51,715 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:51,715 DEBUG [Thread-3028] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:51,715 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:51,715 DEBUG [Thread-3028] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:51,715 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:51,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:51,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:51,715 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:51,815 DEBUG [Thread-3028] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:51,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:51,817 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:51,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:51,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:51,818 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:51,818 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:51,818 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:51,818 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:51,818 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:51,819 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:51,820 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:51,820 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:51,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:51,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:51,820 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:51,820 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:51,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:51,821 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:51,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:51,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:51,821 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:51,821 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:55:51,821 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:51,821 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:51,821 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:51,821 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:51,821 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:51,821 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:51,822 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:51,822 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. started... 2023-04-19 20:55:51,822 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. as already flushing 2023-04-19 20:55:51,822 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:51,822 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:51,822 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:51,822 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:51,822 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:51,822 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:51,823 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:51,823 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:51,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:51,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:51,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:51,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:51,824 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:51,824 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:51,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:51,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:51,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:51,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:51,825 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:51,825 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3ec4360d[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:51,825 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:51,825 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:51,826 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:51,826 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:51,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:51,826 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:51,826 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:51,827 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:51,827 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:51,827 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:51,829 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:51,829 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:51,829 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:51,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:51,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:51,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:51,829 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:51,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:51,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:51,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:51,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:51,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:51,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:51,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:51,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:51,832 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:51,832 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:51,832 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:51,832 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:51,832 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:51,832 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:51,833 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:51,833 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:51,833 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:51,833 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:51,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:51,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:51,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:51,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:51,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:51,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:51,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:51,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:51,835 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:51,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:51,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:51,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:51,838 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:51,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:51,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:51,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:51,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:51,840 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:51,840 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:51,840 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:51,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:51,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:51,841 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:51,841 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:51,841 DEBUG [Thread-3028] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:51,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:51,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:51,841 DEBUG [Thread-3028] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:51,841 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:51,841 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:51,842 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:51,842 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:51,842 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:51,842 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:51,842 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:51,842 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:55:51,843 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:51,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35470 deadline: 1681937811839, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:51,844 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:51,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35486 deadline: 1681937811841, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:51,914 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=701 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/C/e4141273854e4a3a9a931efc95e8a83d 2023-04-19 20:55:51,918 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/A/15792f21d70c4057826d9060c2ec1bb6 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/15792f21d70c4057826d9060c2ec1bb6 2023-04-19 20:55:51,922 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/15792f21d70c4057826d9060c2ec1bb6, entries=200, sequenceid=701, filesize=14.3 K 2023-04-19 20:55:51,922 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/B/9739c501febe47e99417caa65c62a5fb as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/9739c501febe47e99417caa65c62a5fb 2023-04-19 20:55:51,926 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/9739c501febe47e99417caa65c62a5fb, entries=150, sequenceid=701, filesize=11.9 K 2023-04-19 20:55:51,926 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/C/e4141273854e4a3a9a931efc95e8a83d as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/e4141273854e4a3a9a931efc95e8a83d 2023-04-19 20:55:51,930 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/e4141273854e4a3a9a931efc95e8a83d, entries=150, sequenceid=701, filesize=11.9 K 2023-04-19 20:55:51,930 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 4393734ae917e967cf2b5b4b3b44ed0d in 470ms, sequenceid=701, compaction requested=true 2023-04-19 20:55:51,931 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 4393734ae917e967cf2b5b4b3b44ed0d: 2023-04-19 20:55:51,931 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 20:55:51,931 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-19 20:55:51,931 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-19 20:55:51,931 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 20:55:51,931 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 20:55:51,932 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39850 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-19 20:55:51,932 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37410 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-19 20:55:51,932 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1912): 4393734ae917e967cf2b5b4b3b44ed0d/A is initiating minor compaction (all files) 2023-04-19 20:55:51,932 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1912): 4393734ae917e967cf2b5b4b3b44ed0d/B is initiating minor compaction (all files) 2023-04-19 20:55:51,932 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 4393734ae917e967cf2b5b4b3b44ed0d/A in TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:51,932 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4393734ae917e967cf2b5b4b3b44ed0d/B in TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:51,932 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/2eb1ff62a6064687a8818a54b4ce106b, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/2762c960aec24d86bfb7c0a075a48a78, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/15792f21d70c4057826d9060c2ec1bb6] into tmpdir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp, totalSize=38.9 K 2023-04-19 20:55:51,932 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/5aef647ab52c412fb92806e34a67b515, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/4ac0391327d644b2b7c688dbbce1f03e, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/9739c501febe47e99417caa65c62a5fb] into tmpdir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp, totalSize=36.5 K 2023-04-19 20:55:51,933 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 2eb1ff62a6064687a8818a54b4ce106b, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=651, earliestPutTs=1681937749190 2023-04-19 20:55:51,933 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 5aef647ab52c412fb92806e34a67b515, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=651, earliestPutTs=1681937749190 2023-04-19 20:55:51,933 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 2762c960aec24d86bfb7c0a075a48a78, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=688, earliestPutTs=1681937750542 2023-04-19 20:55:51,933 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 4ac0391327d644b2b7c688dbbce1f03e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=688, earliestPutTs=1681937750542 2023-04-19 20:55:51,933 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 15792f21d70c4057826d9060c2ec1bb6, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=701, earliestPutTs=1681937751106 2023-04-19 20:55:51,933 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 9739c501febe47e99417caa65c62a5fb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=701, earliestPutTs=1681937751106 2023-04-19 20:55:51,941 DEBUG [Thread-3028] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:51,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:51,944 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:51,945 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] throttle.PressureAwareThroughputController(145): 4393734ae917e967cf2b5b4b3b44ed0d#A#compaction#467 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 20:55:51,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:51,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:51,948 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:51,948 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:51,948 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:51,948 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:51,949 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:51,949 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:51,949 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] throttle.PressureAwareThroughputController(145): 4393734ae917e967cf2b5b4b3b44ed0d#B#compaction#468 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 20:55:51,950 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:51,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:51,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:51,950 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:51,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:51,950 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:51,950 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:51,951 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:51,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:51,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:51,951 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:51,956 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:55:51,956 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:51,956 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:51,956 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:51,956 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:51,968 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:51,968 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:51,968 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:51,968 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. started... 2023-04-19 20:55:51,968 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4393734ae917e967cf2b5b4b3b44ed0d 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-19 20:55:51,968 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4393734ae917e967cf2b5b4b3b44ed0d, store=A 2023-04-19 20:55:51,969 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:55:51,969 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4393734ae917e967cf2b5b4b3b44ed0d, store=B 2023-04-19 20:55:51,969 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:55:51,969 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4393734ae917e967cf2b5b4b3b44ed0d, store=C 2023-04-19 20:55:51,969 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:55:51,981 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=721 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/A/8e8e5ad0d2b44d9bb3977c9daf67108d 2023-04-19 20:55:52,002 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=721 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/B/7ac6539cfbcd474181bcd610158a054f 2023-04-19 20:55:52,026 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=721 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/C/f18549fccb9f478c925b497a8c3a6513 2023-04-19 20:55:52,031 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/A/8e8e5ad0d2b44d9bb3977c9daf67108d as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/8e8e5ad0d2b44d9bb3977c9daf67108d 2023-04-19 20:55:52,035 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/8e8e5ad0d2b44d9bb3977c9daf67108d, entries=150, sequenceid=721, filesize=11.9 K 2023-04-19 20:55:52,036 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/B/7ac6539cfbcd474181bcd610158a054f as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/7ac6539cfbcd474181bcd610158a054f 2023-04-19 20:55:52,042 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/7ac6539cfbcd474181bcd610158a054f, entries=150, sequenceid=721, filesize=11.9 K 2023-04-19 20:55:52,043 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/C/f18549fccb9f478c925b497a8c3a6513 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/f18549fccb9f478c925b497a8c3a6513 2023-04-19 20:55:52,046 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/f18549fccb9f478c925b497a8c3a6513, entries=150, sequenceid=721, filesize=11.9 K 2023-04-19 20:55:52,046 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=0 B/0 for 4393734ae917e967cf2b5b4b3b44ed0d in 78ms, sequenceid=721, compaction requested=true 2023-04-19 20:55:52,046 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4393734ae917e967cf2b5b4b3b44ed0d: 2023-04-19 20:55:52,047 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:52,047 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:52,047 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:52,047 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:52,047 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:52,047 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:52,050 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:52,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:52,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:52,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:52,050 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:52,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:52,050 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:52,050 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:52,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:52,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:52,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:52,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:52,053 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:52,053 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2e863137[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:52,053 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:52,053 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:52,054 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:52,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:52,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:52,054 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:52,054 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:52,054 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:52,055 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:52,055 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:52,056 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:52,056 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:52,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:52,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:52,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:52,056 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:52,056 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:52,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:52,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:52,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:52,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:52,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:52,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:52,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:52,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:52,058 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:52,058 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:52,059 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:52,059 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:52,059 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:52,059 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:52,060 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:52,060 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:52,060 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:52,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:52,061 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:52,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:52,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:52,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:52,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:52,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:52,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:52,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:52,062 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:52,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:52,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:52,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:52,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:52,063 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:52,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:52,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:52,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:52,066 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:52,066 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:52,066 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:52,066 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:52,067 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:55:52,067 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:52,066 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:52,067 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:52,067 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:52,067 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:52,067 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:52,067 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:52,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:52,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:52,067 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:52,067 DEBUG [Thread-3028] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:52,067 DEBUG [Thread-3028] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:52,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:52,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:52,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(9158): Flush requested on 4393734ae917e967cf2b5b4b3b44ed0d 2023-04-19 20:55:52,165 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 4393734ae917e967cf2b5b4b3b44ed0d 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-19 20:55:52,165 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4393734ae917e967cf2b5b4b3b44ed0d, store=A 2023-04-19 20:55:52,165 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:55:52,165 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4393734ae917e967cf2b5b4b3b44ed0d, store=B 2023-04-19 20:55:52,165 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:55:52,165 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4393734ae917e967cf2b5b4b3b44ed0d, store=C 2023-04-19 20:55:52,165 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:55:52,167 DEBUG [Thread-3028] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:52,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:52,169 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:52,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:52,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:52,170 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:52,170 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:52,170 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:52,170 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:52,171 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:52,171 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:52,174 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:52,174 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:52,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:52,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:52,175 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:52,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:52,175 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:52,175 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:52,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:52,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:52,176 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:52,176 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:55:52,176 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:52,176 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:52,176 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:52,177 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:52,177 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:52,177 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:52,177 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:52,177 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. started... 2023-04-19 20:55:52,177 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. as already flushing 2023-04-19 20:55:52,177 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:52,177 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:52,177 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:52,177 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:52,177 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:52,177 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:52,179 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:52,179 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:52,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:52,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:52,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:52,179 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:52,179 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:52,179 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=732 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/A/49b6d93777064bd389c5e304363837e5 2023-04-19 20:55:52,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:52,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:52,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:52,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:52,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:52,181 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:52,181 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@722a721d[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:52,181 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:52,181 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:52,183 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:52,183 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:52,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:52,183 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:52,183 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:52,183 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:52,183 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:52,183 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:52,188 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:52,188 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:52,188 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:52,188 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:52,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:52,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:52,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:52,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:52,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:52,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:52,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:52,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:52,191 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=732 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/B/2fd9720af527423698fa9ed7a9d2299c 2023-04-19 20:55:52,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:52,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:52,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:52,192 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:52,192 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:52,192 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:52,192 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:52,192 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:52,192 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:52,194 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:52,194 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:52,194 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:52,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:52,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:52,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:52,194 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:52,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:52,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:52,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:52,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:52,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:52,195 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:52,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:52,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:52,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:52,197 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:52,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:52,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:52,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:52,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:52,199 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:52,199 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:52,199 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:52,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:52,200 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:52,200 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:52,200 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:52,199 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:52,200 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:52,200 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:52,199 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:52,200 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:52,200 DEBUG [Thread-3028] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:52,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:52,200 DEBUG [Thread-3028] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:52,200 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:52,200 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:55:52,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:52,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:52,266 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:52,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35486 deadline: 1681937812256, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:52,267 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681937752230, max timestamp=1681937752265Num uniques 859; 2023-04-19 20:55:52,267 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 20:55:52,267 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681937752230, max timestamp=1681937752265Num uniques 859; 2023-04-19 20:55:52,267 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681937752230, max timestamp=1681937752265Num uniques 859; 2023-04-19 20:55:52,268 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 20:55:52,268 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 20:55:52,300 DEBUG [Thread-3028] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:52,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:52,302 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:52,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:52,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:52,303 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:52,303 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:52,303 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:52,303 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:52,304 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:52,304 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:52,305 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:52,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:52,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:52,306 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:52,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:52,306 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:52,306 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:52,306 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:52,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:52,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:52,307 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:52,307 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:55:52,307 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:52,307 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:52,307 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:52,307 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:52,307 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:52,307 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:52,307 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:52,308 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. started... 2023-04-19 20:55:52,308 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. as already flushing 2023-04-19 20:55:52,308 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:52,308 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:52,308 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:52,308 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:52,308 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:52,308 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:52,316 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:52,316 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:52,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:52,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:52,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:52,316 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:52,316 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:52,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:52,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:52,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:52,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:52,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:52,318 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:52,318 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7e75c714[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:52,318 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:52,318 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:52,321 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:52,321 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:52,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:52,321 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:52,321 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:52,321 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:52,321 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:52,321 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:52,323 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:52,323 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:52,323 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:52,323 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:52,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:52,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:52,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:52,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:52,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:52,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:52,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:52,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:52,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:52,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:52,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:52,326 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:52,326 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:52,326 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:52,326 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:52,326 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:52,326 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:52,328 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:52,328 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:52,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:52,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:52,328 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:52,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:52,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:52,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:52,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:52,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:52,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:52,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:52,329 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:52,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:52,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:52,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:52,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:52,330 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:52,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:52,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:52,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:52,332 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:52,333 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:52,333 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:55:52,333 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:52,333 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:52,333 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:52,333 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:52,333 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:52,333 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:52,333 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:52,333 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:52,333 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:52,333 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:52,333 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:52,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:52,333 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:52,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:52,335 DEBUG [Thread-3028] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:52,335 DEBUG [Thread-3028] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:52,370 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/A/511e30f9f46d4d86aae5ae1b9d22e146 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/511e30f9f46d4d86aae5ae1b9d22e146 2023-04-19 20:55:52,373 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 4393734ae917e967cf2b5b4b3b44ed0d/A of 4393734ae917e967cf2b5b4b3b44ed0d into 511e30f9f46d4d86aae5ae1b9d22e146(size=12.8 K), total size for store is 24.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 20:55:52,373 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4393734ae917e967cf2b5b4b3b44ed0d: 2023-04-19 20:55:52,373 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d., storeName=4393734ae917e967cf2b5b4b3b44ed0d/A, priority=13, startTime=1681937751931; duration=0sec 2023-04-19 20:55:52,374 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 20:55:52,374 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-19 20:55:52,376 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49596 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-19 20:55:52,376 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1912): 4393734ae917e967cf2b5b4b3b44ed0d/C is initiating minor compaction (all files) 2023-04-19 20:55:52,376 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 4393734ae917e967cf2b5b4b3b44ed0d/C in TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:52,376 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/47fd38401d6e4e24bdc64b779e839c09, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/e823084c3f004185a2ef1ced67cafc9a, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/e4141273854e4a3a9a931efc95e8a83d, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/f18549fccb9f478c925b497a8c3a6513] into tmpdir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp, totalSize=48.4 K 2023-04-19 20:55:52,377 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 47fd38401d6e4e24bdc64b779e839c09, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=651, earliestPutTs=1681937749190 2023-04-19 20:55:52,377 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting e823084c3f004185a2ef1ced67cafc9a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=688, earliestPutTs=1681937750542 2023-04-19 20:55:52,378 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting e4141273854e4a3a9a931efc95e8a83d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=701, earliestPutTs=1681937751106 2023-04-19 20:55:52,378 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting f18549fccb9f478c925b497a8c3a6513, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=721, earliestPutTs=1681937751519 2023-04-19 20:55:52,380 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/B/192809bb0249425883b8a36031c86f31 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/192809bb0249425883b8a36031c86f31 2023-04-19 20:55:52,384 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 4393734ae917e967cf2b5b4b3b44ed0d/B of 4393734ae917e967cf2b5b4b3b44ed0d into 192809bb0249425883b8a36031c86f31(size=12.8 K), total size for store is 24.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 20:55:52,384 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4393734ae917e967cf2b5b4b3b44ed0d: 2023-04-19 20:55:52,384 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d., storeName=4393734ae917e967cf2b5b4b3b44ed0d/B, priority=13, startTime=1681937751931; duration=0sec 2023-04-19 20:55:52,384 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 20:55:52,386 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] throttle.PressureAwareThroughputController(145): 4393734ae917e967cf2b5b4b3b44ed0d#C#compaction#475 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 20:55:52,389 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:52,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35470 deadline: 1681937812387, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:52,399 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:52,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35486 deadline: 1681937812390, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:52,435 DEBUG [Thread-3028] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:52,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:52,437 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:52,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:52,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:52,438 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:52,438 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:52,438 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:52,438 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:52,439 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:52,439 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:52,440 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:52,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:52,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:52,440 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:52,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:52,440 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:52,441 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:52,441 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:52,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:52,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:52,441 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:52,441 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:55:52,441 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:52,441 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:52,441 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:52,441 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:52,442 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:52,442 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:52,442 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:52,442 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. started... 2023-04-19 20:55:52,442 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. as already flushing 2023-04-19 20:55:52,442 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:52,442 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:52,442 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:52,442 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:52,442 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:52,442 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:52,444 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:52,444 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:52,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:52,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:52,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:52,444 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:52,444 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:52,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:52,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:52,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:52,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:52,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:52,445 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:52,445 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@611ef289[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:52,445 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:52,446 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:52,453 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:52,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:52,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:52,454 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:52,454 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:52,454 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:52,454 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:52,454 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:52,455 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:52,455 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:52,455 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:52,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:52,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:52,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:52,455 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:52,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:52,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:52,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:52,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:52,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:52,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:52,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:52,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:52,459 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:52,459 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:52,460 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:52,460 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:52,460 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:52,460 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:52,461 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:52,461 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:52,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:52,461 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:52,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:52,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:52,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:52,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:52,462 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:52,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:52,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:52,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:52,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:52,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:52,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:52,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:52,463 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:52,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:52,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:52,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:52,467 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:52,467 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:52,467 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:52,467 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:52,467 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:52,467 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:52,467 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:52,467 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:52,467 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:52,467 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:52,467 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:52,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:52,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:52,467 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:52,468 DEBUG [Thread-3028] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:52,468 DEBUG [Thread-3028] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:52,468 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:55:52,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:52,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:52,495 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:52,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35470 deadline: 1681937812491, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:52,506 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:52,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35486 deadline: 1681937812500, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:52,568 DEBUG [Thread-3028] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:52,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:52,570 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:52,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:52,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:52,571 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:52,571 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:52,571 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:52,571 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:52,571 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:52,571 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:52,577 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:52,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:52,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:52,577 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:52,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:52,577 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:52,577 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:52,577 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:52,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:52,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:52,578 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:52,578 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:55:52,578 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:52,578 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:52,578 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:52,578 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:52,578 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:52,578 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:52,578 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:52,578 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. started... 2023-04-19 20:55:52,578 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. as already flushing 2023-04-19 20:55:52,579 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:52,579 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:52,579 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:52,579 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:52,579 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:52,579 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:52,582 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:52,582 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:52,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:52,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:52,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:52,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:52,583 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:52,583 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:52,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:52,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:52,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:52,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:52,584 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:52,584 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7e0e6814[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:52,584 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:52,584 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:52,585 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:52,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:52,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:52,585 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:52,585 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:52,585 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:52,586 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:52,586 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:52,587 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:52,587 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:52,587 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:52,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:52,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:52,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:52,587 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:52,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:52,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:52,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:52,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:52,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:52,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:52,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:52,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:52,590 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:52,590 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:52,590 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:52,590 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:52,590 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:52,590 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:52,593 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:52,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:52,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:52,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:52,593 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:52,593 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:52,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:52,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:52,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:52,593 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:52,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:52,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:52,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:52,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:52,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:52,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:52,594 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:52,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:52,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:52,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:52,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:52,596 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:52,596 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:52,596 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:52,596 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:52,596 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:52,596 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:52,597 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:52,597 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:52,597 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:55:52,597 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:52,597 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:52,597 DEBUG [Thread-3028] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:52,598 DEBUG [Thread-3028] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:52,596 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:52,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:52,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:52,598 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:52,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:52,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:52,606 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=732 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/C/3e96a02b88904cb09ee3b3488a81a159 2023-04-19 20:55:52,611 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/A/49b6d93777064bd389c5e304363837e5 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/49b6d93777064bd389c5e304363837e5 2023-04-19 20:55:52,614 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/49b6d93777064bd389c5e304363837e5, entries=100, sequenceid=732, filesize=9.5 K 2023-04-19 20:55:52,615 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/B/2fd9720af527423698fa9ed7a9d2299c as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/2fd9720af527423698fa9ed7a9d2299c 2023-04-19 20:55:52,618 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/2fd9720af527423698fa9ed7a9d2299c, entries=100, sequenceid=732, filesize=9.5 K 2023-04-19 20:55:52,619 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/C/3e96a02b88904cb09ee3b3488a81a159 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/3e96a02b88904cb09ee3b3488a81a159 2023-04-19 20:55:52,622 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/3e96a02b88904cb09ee3b3488a81a159, entries=100, sequenceid=732, filesize=9.5 K 2023-04-19 20:55:52,622 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 4393734ae917e967cf2b5b4b3b44ed0d in 457ms, sequenceid=732, compaction requested=true 2023-04-19 20:55:52,622 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 4393734ae917e967cf2b5b4b3b44ed0d: 2023-04-19 20:55:52,623 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 20:55:52,623 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 20:55:52,623 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-19 20:55:52,623 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-19 20:55:52,624 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35068 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-19 20:55:52,624 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1912): 4393734ae917e967cf2b5b4b3b44ed0d/A is initiating minor compaction (all files) 2023-04-19 20:55:52,624 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4393734ae917e967cf2b5b4b3b44ed0d/A in TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:52,624 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/511e30f9f46d4d86aae5ae1b9d22e146, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/8e8e5ad0d2b44d9bb3977c9daf67108d, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/49b6d93777064bd389c5e304363837e5] into tmpdir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp, totalSize=34.2 K 2023-04-19 20:55:52,624 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 511e30f9f46d4d86aae5ae1b9d22e146, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=701, earliestPutTs=1681937751106 2023-04-19 20:55:52,624 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 8e8e5ad0d2b44d9bb3977c9daf67108d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=721, earliestPutTs=1681937751519 2023-04-19 20:55:52,625 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 49b6d93777064bd389c5e304363837e5, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=732, earliestPutTs=1681937752164 2023-04-19 20:55:52,630 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] throttle.PressureAwareThroughputController(145): 4393734ae917e967cf2b5b4b3b44ed0d#A#compaction#476 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 20:55:52,638 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/A/95eee51867de4585b5da7fe182f100c3 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/95eee51867de4585b5da7fe182f100c3 2023-04-19 20:55:52,642 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 4393734ae917e967cf2b5b4b3b44ed0d/A of 4393734ae917e967cf2b5b4b3b44ed0d into 95eee51867de4585b5da7fe182f100c3(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 20:55:52,642 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4393734ae917e967cf2b5b4b3b44ed0d: 2023-04-19 20:55:52,642 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d., storeName=4393734ae917e967cf2b5b4b3b44ed0d/A, priority=13, startTime=1681937752623; duration=0sec 2023-04-19 20:55:52,642 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-19 20:55:52,643 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 4 compacting, 1 eligible, 16 blocking 2023-04-19 20:55:52,643 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-19 20:55:52,643 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-19 20:55:52,643 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. because compaction request was cancelled 2023-04-19 20:55:52,643 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-19 20:55:52,644 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35068 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-19 20:55:52,644 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1912): 4393734ae917e967cf2b5b4b3b44ed0d/B is initiating minor compaction (all files) 2023-04-19 20:55:52,644 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4393734ae917e967cf2b5b4b3b44ed0d/B in TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:52,644 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/192809bb0249425883b8a36031c86f31, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/7ac6539cfbcd474181bcd610158a054f, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/2fd9720af527423698fa9ed7a9d2299c] into tmpdir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp, totalSize=34.2 K 2023-04-19 20:55:52,645 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 192809bb0249425883b8a36031c86f31, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=701, earliestPutTs=1681937751106 2023-04-19 20:55:52,645 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 7ac6539cfbcd474181bcd610158a054f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=721, earliestPutTs=1681937751519 2023-04-19 20:55:52,645 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 2fd9720af527423698fa9ed7a9d2299c, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=732, earliestPutTs=1681937752164 2023-04-19 20:55:52,651 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] throttle.PressureAwareThroughputController(145): 4393734ae917e967cf2b5b4b3b44ed0d#B#compaction#477 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 20:55:52,698 DEBUG [Thread-3028] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:52,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:52,700 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:52,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:52,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(9158): Flush requested on 4393734ae917e967cf2b5b4b3b44ed0d 2023-04-19 20:55:52,701 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 4393734ae917e967cf2b5b4b3b44ed0d 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-19 20:55:52,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:52,702 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:52,702 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4393734ae917e967cf2b5b4b3b44ed0d, store=A 2023-04-19 20:55:52,702 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:52,702 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 20:55:52,702 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:52,702 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:52,702 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4393734ae917e967cf2b5b4b3b44ed0d, store=B 2023-04-19 20:55:52,702 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 20:55:52,702 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4393734ae917e967cf2b5b4b3b44ed0d, store=C 2023-04-19 20:55:52,702 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 20:55:52,702 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:52,702 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:52,704 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:52,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:52,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:52,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:52,705 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:52,705 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:52,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:52,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:52,706 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:52,706 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:52,706 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:52,706 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:55:52,706 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:52,706 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:52,706 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:52,706 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:52,706 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:52,706 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:52,706 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:52,706 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. started... 2023-04-19 20:55:52,707 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. as already flushing 2023-04-19 20:55:52,707 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:52,707 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:52,707 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:52,707 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:52,707 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:52,707 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:52,709 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:52,709 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:52,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:52,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:52,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:52,709 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=772 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/A/3a6b8198c56047fa9f55bce050f5eb83 2023-04-19 20:55:52,709 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:52,709 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:52,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:52,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:52,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:52,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:52,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:52,712 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:52,712 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5f155752[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:52,712 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:52,712 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:52,714 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:52,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:52,714 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:52,714 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:52,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:52,714 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:52,714 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:52,714 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:52,716 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:52,716 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:52,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:52,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:52,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:52,716 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:52,716 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:52,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:52,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:52,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:52,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:52,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:52,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:52,719 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=772 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/B/635db6a1d809458d80e73e3b897cf07b 2023-04-19 20:55:52,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:52,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:52,721 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:52,721 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:52,721 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:52,721 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:52,721 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:52,721 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:52,722 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:52,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 251 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35490 deadline: 1681937812712, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:52,722 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:52,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35470 deadline: 1681937812713, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:52,725 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:52,725 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:52,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:52,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:52,725 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:52,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:52,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:52,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:52,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:52,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:52,726 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:52,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:52,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:52,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:52,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:52,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:52,728 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:52,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:52,729 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:52,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:52,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 256 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35474 deadline: 1681937812719, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:52,729 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:52,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35486 deadline: 1681937812720, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:52,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:52,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:52,730 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:52,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35492 deadline: 1681937812721, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:52,731 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:52,731 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:52,731 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:52,731 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:52,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:52,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:52,731 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:52,731 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:52,731 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:52,731 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:52,731 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:52,731 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:52,731 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:52,731 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:52,731 DEBUG [Thread-3028] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:52,731 DEBUG [Thread-3028] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:52,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:52,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:52,732 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:55:52,733 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=772 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/C/fedbba22989346bd9f9d053a55b21d12 2023-04-19 20:55:52,736 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/A/3a6b8198c56047fa9f55bce050f5eb83 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/3a6b8198c56047fa9f55bce050f5eb83 2023-04-19 20:55:52,739 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/3a6b8198c56047fa9f55bce050f5eb83, entries=200, sequenceid=772, filesize=14.3 K 2023-04-19 20:55:52,740 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/B/635db6a1d809458d80e73e3b897cf07b as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/635db6a1d809458d80e73e3b897cf07b 2023-04-19 20:55:52,743 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/635db6a1d809458d80e73e3b897cf07b, entries=150, sequenceid=772, filesize=11.9 K 2023-04-19 20:55:52,745 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/C/fedbba22989346bd9f9d053a55b21d12 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/fedbba22989346bd9f9d053a55b21d12 2023-04-19 20:55:52,748 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/fedbba22989346bd9f9d053a55b21d12, entries=150, sequenceid=772, filesize=11.9 K 2023-04-19 20:55:52,749 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for 4393734ae917e967cf2b5b4b3b44ed0d in 48ms, sequenceid=772, compaction requested=false 2023-04-19 20:55:52,749 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 4393734ae917e967cf2b5b4b3b44ed0d: 2023-04-19 20:55:52,798 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/C/e7c8e6b0fa934031a1395dcdc0c697e6 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/e7c8e6b0fa934031a1395dcdc0c697e6 2023-04-19 20:55:52,802 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 4393734ae917e967cf2b5b4b3b44ed0d/C of 4393734ae917e967cf2b5b4b3b44ed0d into e7c8e6b0fa934031a1395dcdc0c697e6(size=12.9 K), total size for store is 34.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 20:55:52,802 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4393734ae917e967cf2b5b4b3b44ed0d: 2023-04-19 20:55:52,803 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d., storeName=4393734ae917e967cf2b5b4b3b44ed0d/C, priority=12, startTime=1681937751931; duration=0sec 2023-04-19 20:55:52,803 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 20:55:52,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(9158): Flush requested on 4393734ae917e967cf2b5b4b3b44ed0d 2023-04-19 20:55:52,826 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 4393734ae917e967cf2b5b4b3b44ed0d 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-19 20:55:52,826 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4393734ae917e967cf2b5b4b3b44ed0d, store=A 2023-04-19 20:55:52,826 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:55:52,826 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4393734ae917e967cf2b5b4b3b44ed0d, store=B 2023-04-19 20:55:52,826 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:55:52,826 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4393734ae917e967cf2b5b4b3b44ed0d, store=C 2023-04-19 20:55:52,826 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:55:52,832 DEBUG [Thread-3028] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:52,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:52,834 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:52,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:52,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:52,835 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:52,835 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:52,835 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:52,835 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:52,836 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:52,836 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:52,838 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:52,838 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:52,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:52,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:52,838 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:52,838 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:52,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:52,838 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:52,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:52,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:52,839 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:52,839 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:55:52,839 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:52,839 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:52,839 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:52,839 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:52,839 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:52,839 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:52,839 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:52,839 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. started... 2023-04-19 20:55:52,839 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. as already flushing 2023-04-19 20:55:52,839 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:52,839 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:52,839 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:52,839 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:52,839 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:52,839 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:52,841 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:52,841 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:52,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:52,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:52,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:52,841 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:52,841 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:52,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:52,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:52,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:52,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:52,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:52,842 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:52,842 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@f0876c3[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:52,842 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:52,842 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:52,843 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:52,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:52,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:52,843 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:52,843 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:52,843 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:52,843 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:52,843 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:52,845 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:52,845 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:52,845 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:52,846 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:52,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:52,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:52,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:52,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:52,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:52,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:52,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:52,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:52,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:52,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:52,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:52,849 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:52,849 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:52,849 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:52,849 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:52,849 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:52,849 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:52,852 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:52,852 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:52,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:52,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:52,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:52,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:52,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:52,852 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:52,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:52,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:52,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:52,852 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:52,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:52,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:52,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:52,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:52,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:52,853 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:52,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:52,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:52,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:52,856 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:52,856 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:52,856 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:52,856 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:52,856 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:52,856 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:52,856 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:52,856 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:55:52,856 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:52,856 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:52,856 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:52,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:52,856 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:52,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:52,856 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:52,857 DEBUG [Thread-3028] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:52,857 DEBUG [Thread-3028] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:52,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:52,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:52,892 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:52,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35470 deadline: 1681937812882, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:52,893 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:52,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 261 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35474 deadline: 1681937812883, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:52,893 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:52,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 235 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35492 deadline: 1681937812886, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:52,893 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681937752858, max timestamp=1681937752891Num uniques 859; 2023-04-19 20:55:52,893 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-19 20:55:52,893 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681937752858, max timestamp=1681937752891Num uniques 859; 2023-04-19 20:55:52,893 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681937752858, max timestamp=1681937752891Num uniques 859; 2023-04-19 20:55:52,893 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-19 20:55:52,894 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-19 20:55:52,957 DEBUG [Thread-3028] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:52,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:52,959 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:52,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:52,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:52,960 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:52,960 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:52,960 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:52,960 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:52,960 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:52,960 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:52,961 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:52,961 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:52,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:52,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:52,962 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:52,962 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:52,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:52,962 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:52,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:52,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:52,962 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:52,963 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:55:52,963 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:52,963 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:52,963 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:52,963 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:52,963 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:52,963 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:52,963 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:52,963 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. started... 2023-04-19 20:55:52,963 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. as already flushing 2023-04-19 20:55:52,963 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:52,963 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:52,963 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:52,963 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:52,963 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:52,963 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:52,965 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:52,965 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:52,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:52,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:52,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:52,965 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:52,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:52,965 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:52,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:52,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:52,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:52,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:52,966 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:52,966 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@19944711[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:52,966 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:52,966 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:52,969 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:52,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:52,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:52,969 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:52,969 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:52,969 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:52,969 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:52,969 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:52,970 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:52,970 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:52,970 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:52,970 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:52,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:52,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:52,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:52,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:52,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:52,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:52,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:52,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:52,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:52,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:52,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:52,972 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:52,972 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:52,972 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:52,972 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:52,973 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:52,973 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:52,974 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:52,974 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:52,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:52,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:52,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:52,974 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:52,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:52,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:52,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:52,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:52,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:52,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:52,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:52,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:52,975 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:52,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:52,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:52,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:52,977 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:52,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:52,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:52,977 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:52,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 265 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35490 deadline: 1681937812968, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:52,979 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:52,979 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:52,979 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:52,979 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:52,979 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:52,979 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:52,979 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:52,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:52,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:52,979 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:52,979 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:52,979 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:52,979 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:52,979 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:52,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:52,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:52,979 DEBUG [Thread-3028] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:52,980 DEBUG [Thread-3028] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:52,985 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:52,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35486 deadline: 1681937812978, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:52,997 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:52,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35470 deadline: 1681937812993, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:53,000 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:53,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 263 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35474 deadline: 1681937812994, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:53,000 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:53,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 237 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35492 deadline: 1681937812994, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:53,062 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/B/8550f0c81b7046baacb90f80db34e2f3 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/8550f0c81b7046baacb90f80db34e2f3 2023-04-19 20:55:53,066 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 4393734ae917e967cf2b5b4b3b44ed0d/B of 4393734ae917e967cf2b5b4b3b44ed0d into 8550f0c81b7046baacb90f80db34e2f3(size=12.9 K), total size for store is 24.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 20:55:53,066 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4393734ae917e967cf2b5b4b3b44ed0d: 2023-04-19 20:55:53,066 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d., storeName=4393734ae917e967cf2b5b4b3b44ed0d/B, priority=13, startTime=1681937752623; duration=0sec 2023-04-19 20:55:53,066 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 20:55:53,080 DEBUG [Thread-3028] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:53,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:53,082 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:53,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:53,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:53,083 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:53,083 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:53,083 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:53,083 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:53,083 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:53,083 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:53,084 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:53,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 267 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35490 deadline: 1681937813079, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:53,085 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:53,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:53,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:53,085 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:53,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:53,086 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:53,086 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:53,086 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:53,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:53,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:53,087 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:53,087 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:55:53,087 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:53,087 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:53,087 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:53,087 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:53,087 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:53,087 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:53,087 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:53,087 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. started... 2023-04-19 20:55:53,087 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. as already flushing 2023-04-19 20:55:53,088 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:53,088 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:53,088 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:53,088 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:53,088 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:53,088 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:53,089 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:53,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:53,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:53,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:53,090 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:53,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:53,090 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:53,090 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:53,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:53,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:53,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:53,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:53,091 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:53,091 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1457c2ff[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:53,091 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:53,092 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:53,092 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:53,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35486 deadline: 1681937813086, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:53,094 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:53,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:53,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:53,094 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:53,094 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:53,094 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:53,094 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:53,094 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:53,095 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:53,095 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:53,095 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:53,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:53,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:53,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:53,095 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:53,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:53,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:53,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:53,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:53,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:53,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:53,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:53,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:53,098 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:53,098 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:53,098 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:53,098 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:53,098 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:53,098 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:53,108 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:53,108 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:53,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:53,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:53,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:53,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:53,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:53,108 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:53,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:53,109 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:53,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:53,109 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:53,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:53,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:53,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:53,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:53,110 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:53,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:53,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:53,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:53,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:53,114 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:53,114 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:53,114 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:53,114 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:53,114 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:53,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:53,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:53,114 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:53,114 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:53,114 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:55:53,114 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:53,115 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:53,115 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:53,115 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:53,115 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:53,115 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:53,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:53,115 DEBUG [Thread-3028] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:53,115 DEBUG [Thread-3028] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:53,203 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:53,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35470 deadline: 1681937813198, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:53,207 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:53,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35492 deadline: 1681937813201, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:53,208 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:53,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 265 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35474 deadline: 1681937813202, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:53,215 DEBUG [Thread-3028] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:53,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:53,217 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:53,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:53,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:53,218 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:53,219 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:53,219 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:53,219 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:53,219 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:53,219 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:53,220 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:53,220 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:53,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:53,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:53,221 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:53,221 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:53,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:53,221 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:53,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:53,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:53,221 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:53,222 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:55:53,222 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:53,222 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:53,222 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:53,222 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:53,222 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:53,222 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:53,222 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:53,222 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. started... 2023-04-19 20:55:53,222 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. as already flushing 2023-04-19 20:55:53,222 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:53,222 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:53,223 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:53,223 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:53,223 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:53,223 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:53,224 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:53,224 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:53,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:53,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:53,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:53,225 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:53,225 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:53,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:53,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:53,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:53,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:53,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:53,226 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:53,226 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@610db8e0[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:53,226 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:53,226 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:53,228 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:53,228 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:53,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:53,228 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:53,228 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:53,228 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:53,228 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:53,228 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:53,230 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:53,230 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:53,230 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:53,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:53,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:53,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:53,230 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:53,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:53,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:53,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:53,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:53,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:53,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:53,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:53,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:53,234 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:53,234 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:53,234 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:53,234 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:53,234 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:53,234 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:53,235 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:53,235 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:53,235 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:53,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:53,235 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:53,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:53,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:53,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:53,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:53,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:53,236 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=784 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/A/d12bcdc968304759b8bd4a6baa702ec0 2023-04-19 20:55:53,236 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:53,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:53,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:53,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:53,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:53,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:53,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:53,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:53,239 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:53,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:53,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:53,243 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:53,243 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:53,243 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:53,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:53,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:53,243 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:53,243 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:53,243 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:53,243 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:53,243 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:53,243 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:53,243 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:53,243 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:53,243 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:53,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:53,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:53,245 DEBUG [Thread-3028] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:53,245 DEBUG [Thread-3028] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:53,291 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:53,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 269 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35490 deadline: 1681937813286, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:53,298 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:53,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35486 deadline: 1681937813294, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:53,345 DEBUG [Thread-3028] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:53,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:53,347 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:53,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:53,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:53,348 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:53,349 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:53,349 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:53,349 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:53,349 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:53,349 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:53,352 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:53,352 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:53,352 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:53,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:53,352 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:53,352 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:53,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:53,353 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:53,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:53,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:53,353 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:53,353 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:55:53,354 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:53,354 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:53,354 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:53,354 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:53,354 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:53,354 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:53,354 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:53,354 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. started... 2023-04-19 20:55:53,354 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. as already flushing 2023-04-19 20:55:53,354 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:53,354 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:53,354 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:53,354 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:53,354 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:53,354 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:53,356 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:53,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:53,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:53,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:53,356 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:53,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:53,356 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:53,356 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:53,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:53,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:53,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:53,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:53,358 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:53,358 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2dff646c[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:53,358 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:53,358 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:53,359 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:53,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:53,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:53,360 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:53,360 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:53,360 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:53,360 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:53,360 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:53,361 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:53,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:53,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:53,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:53,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:53,362 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:53,362 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:53,362 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:53,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:53,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:53,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:53,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:53,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:53,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:53,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:53,365 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:53,365 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:53,365 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:53,365 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:53,365 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:53,365 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:53,367 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:53,367 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:53,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:53,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:53,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:53,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:53,368 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:53,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:53,368 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:53,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:53,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:53,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:53,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:53,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:53,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:53,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:53,369 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:53,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:53,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:53,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:53,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:53,372 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:53,372 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:53,372 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:53,372 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:53,372 DEBUG [Thread-3028] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:53,372 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:55:53,372 DEBUG [Thread-3028] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:53,372 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:53,372 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:53,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:53,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:53,373 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:53,373 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:53,373 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:53,373 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:53,373 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:53,373 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:53,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:53,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:53,473 DEBUG [Thread-3028] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:53,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:53,475 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:53,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:53,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:53,476 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:53,476 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:53,476 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:53,476 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:53,476 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:53,476 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:53,478 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:53,478 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:53,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:53,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:53,478 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:53,478 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:53,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:53,478 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:53,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:53,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:53,479 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:53,479 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:53,479 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:53,479 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:53,479 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:53,479 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:53,479 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:53,479 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:53,479 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. started... 2023-04-19 20:55:53,480 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. as already flushing 2023-04-19 20:55:53,480 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:53,480 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:53,480 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:53,480 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:53,480 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:53,480 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:53,481 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:53,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:53,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:53,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:53,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:53,482 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:53,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:53,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:53,482 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:53,482 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:53,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:53,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:53,483 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:53,483 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@16e8020d[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:53,483 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:53,483 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:53,485 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:53,485 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:53,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:53,485 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:53,485 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:53,485 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:53,485 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:53,485 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:53,486 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:53,486 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:53,486 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:53,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:53,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:53,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:53,487 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:53,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:53,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:53,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:53,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:53,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:53,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:53,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:53,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:53,489 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:53,489 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:53,489 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:53,489 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:53,489 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:53,489 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:53,490 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:53,490 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:53,490 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:53,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:53,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:53,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:53,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:53,491 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:53,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:53,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:53,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:53,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:53,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:53,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:53,492 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:53,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:53,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:53,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:53,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:53,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:53,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:53,498 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:53,498 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:53,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:53,498 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:53,498 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:53,498 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:55:53,498 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:53,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:53,498 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:53,498 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:53,498 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:53,498 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:53,498 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:53,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:53,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:53,499 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:53,499 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:53,499 DEBUG [Thread-3028] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:53,499 DEBUG [Thread-3028] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:53,511 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:53,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35470 deadline: 1681937813506, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:53,517 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:53,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 241 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35492 deadline: 1681937813510, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:53,518 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:53,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 267 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35474 deadline: 1681937813511, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:53,599 DEBUG [Thread-3028] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:53,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:53,601 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:53,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:53,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:53,602 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:53,603 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:53,603 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:53,603 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:53,603 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:53,603 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:53,603 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:53,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 271 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35490 deadline: 1681937813594, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:53,603 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:53,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35486 deadline: 1681937813601, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:53,604 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:53,604 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:53,604 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:53,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:53,605 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:53,605 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:53,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:53,605 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:53,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:53,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:53,605 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:53,606 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:55:53,606 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:53,606 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:53,606 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:53,606 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:53,606 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:53,606 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:53,606 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:53,606 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. started... 2023-04-19 20:55:53,606 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. as already flushing 2023-04-19 20:55:53,606 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:53,606 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:53,606 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:53,606 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:53,606 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:53,606 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:53,608 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:53,608 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:53,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:53,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:53,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:53,608 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:53,608 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:53,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:53,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:53,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:53,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:53,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:53,609 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:53,609 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3c826a15[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:53,609 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:53,609 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:53,611 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:53,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:53,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:53,611 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:53,611 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:53,611 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:53,611 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:53,611 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:53,613 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:53,613 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:53,613 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:53,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:53,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:53,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:53,613 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:53,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:53,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:53,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:53,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:53,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:53,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:53,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:53,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:53,616 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:53,616 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:53,616 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:53,616 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:53,616 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:53,616 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:53,619 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:53,619 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:53,619 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:53,619 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:53,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:53,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:53,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:53,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:53,619 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:53,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:53,620 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:53,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:53,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:53,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:53,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:53,621 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:53,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:53,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:53,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:53,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:53,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:53,623 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:53,623 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:53,623 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:53,623 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:53,623 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:53,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:53,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:53,623 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:53,623 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:53,623 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:53,623 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:53,623 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:53,623 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:53,623 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:53,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:53,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:53,624 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:55:53,624 DEBUG [Thread-3028] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:53,624 DEBUG [Thread-3028] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:53,648 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=784 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/B/619ac4ba267946ecbd77864f8739f7ab 2023-04-19 20:55:53,724 DEBUG [Thread-3028] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:53,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:53,726 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:53,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:53,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:53,727 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:53,727 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:53,727 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:53,727 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:53,728 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:53,728 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:53,729 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:53,729 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:53,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:53,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:53,730 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:53,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:53,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:53,730 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:53,730 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:53,730 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:55:53,730 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:53,731 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:53,731 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:53,731 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:53,731 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:53,731 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:53,731 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:53,731 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:53,731 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:53,731 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. started... 2023-04-19 20:55:53,731 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. as already flushing 2023-04-19 20:55:53,731 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:53,731 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:53,731 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:53,731 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:53,731 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:53,731 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:53,734 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:53,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:53,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:53,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:53,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:53,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:53,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:53,735 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:53,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:53,735 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:53,735 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:53,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:53,736 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:53,736 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@30440fdd[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:53,736 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:53,736 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:53,737 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:53,737 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:53,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:53,737 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:53,737 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:53,737 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:53,737 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:53,738 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:53,739 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:53,739 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:53,739 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:53,739 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:53,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:53,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:53,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:53,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:53,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:53,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:53,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:53,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:53,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:53,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:53,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:53,741 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:53,741 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:53,741 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:53,741 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:53,741 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:53,741 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:53,742 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:53,743 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:53,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:53,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:53,743 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:53,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:53,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:53,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:53,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:53,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:53,743 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:53,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:53,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:53,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:53,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:53,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:53,744 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:53,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:53,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:53,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:53,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:53,749 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:53,749 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:53,749 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:53,749 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:53,749 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:53,749 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:53,749 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:53,750 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:53,749 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:53,749 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:53,749 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:53,750 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:53,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:53,750 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:53,750 DEBUG [Thread-3028] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:53,750 DEBUG [Thread-3028] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:53,750 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:53,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:53,850 DEBUG [Thread-3028] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:53,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:53,852 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:53,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:53,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:53,854 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:53,854 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:53,854 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:53,854 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:53,854 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:53,854 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:53,855 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:53,856 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:53,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:53,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:53,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:53,860 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:53,860 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:53,860 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:53,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:53,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:53,861 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:53,864 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:55:53,864 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:53,864 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:53,864 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:53,864 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:53,864 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:53,864 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:53,864 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:53,864 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. started... 2023-04-19 20:55:53,864 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. as already flushing 2023-04-19 20:55:53,864 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:53,864 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:53,865 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:53,865 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:53,865 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:53,865 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:53,866 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:53,866 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:53,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:53,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:53,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:53,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:53,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:53,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:53,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:53,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:53,868 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:53,868 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:53,868 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:53,868 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@664ec1bd[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:53,868 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:53,868 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:53,870 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:53,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:53,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:53,870 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:53,870 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:53,870 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:53,870 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:53,870 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:53,872 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:53,872 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:53,872 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:53,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:53,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:53,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:53,872 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:53,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:53,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:53,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:53,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:53,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:53,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:53,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:53,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:53,876 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:53,876 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:53,877 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:53,877 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:53,877 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:53,877 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:53,878 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:53,878 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:53,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:53,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:53,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:53,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:53,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:53,878 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:53,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:53,879 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:53,879 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:53,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:53,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:53,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:53,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:53,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:53,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:53,880 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:53,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:53,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:53,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:53,885 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:53,885 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:53,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:53,885 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:53,885 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:53,885 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:53,885 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:53,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:53,885 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:53,885 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:53,885 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:53,885 DEBUG [Thread-3028] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:53,886 DEBUG [Thread-3028] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:53,885 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:53,886 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:53,886 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:53,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:53,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:53,986 DEBUG [Thread-3028] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:53,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:53,988 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:53,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:53,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:53,989 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:53,989 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:53,989 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:53,989 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:53,990 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:53,990 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:53,993 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:53,993 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:53,993 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:53,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:53,993 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:53,993 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:53,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:53,994 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:53,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:53,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:53,994 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:53,995 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:55:53,995 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:53,995 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:53,995 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:53,995 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:53,995 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:53,995 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:53,995 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:53,995 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. started... 2023-04-19 20:55:53,995 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. as already flushing 2023-04-19 20:55:53,995 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:53,995 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:53,995 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:53,995 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:53,996 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:53,996 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:53,999 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:53,999 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:53,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:53,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:53,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:53,999 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:53,999 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:53,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:53,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:54,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:54,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:54,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:54,002 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:54,002 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@30605f13[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:54,002 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:54,002 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:54,003 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:54,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:54,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:54,003 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:54,003 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:54,003 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:54,005 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:54,005 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:54,006 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:54,006 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:54,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:54,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:54,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:54,007 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:54,007 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:54,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:54,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:54,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:54,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:54,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:54,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:54,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:54,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:54,009 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:54,009 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:54,009 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:54,009 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:54,009 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:54,009 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:54,013 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:54,013 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:54,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:54,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:54,013 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:54,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:54,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:54,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:54,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:54,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:54,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:54,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:54,014 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:54,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:54,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:54,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:54,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:54,015 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:54,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:54,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:54,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:54,019 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:54,019 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:54,019 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:54,019 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:54,019 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:54,019 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:54,019 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:54,019 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:54,019 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:54,019 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:54,019 DEBUG [Thread-3028] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:54,019 DEBUG [Thread-3028] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:54,020 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:54,020 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:54,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35470 deadline: 1681937814016, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:54,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:54,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:54,020 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:54,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:54,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:54,027 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:54,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 269 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35474 deadline: 1681937814020, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:54,027 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:54,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 243 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35492 deadline: 1681937814021, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:54,057 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=784 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/C/8b0eefd0196a45b8b810ca16eee6bb25 2023-04-19 20:55:54,061 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/A/d12bcdc968304759b8bd4a6baa702ec0 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/d12bcdc968304759b8bd4a6baa702ec0 2023-04-19 20:55:54,066 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/d12bcdc968304759b8bd4a6baa702ec0, entries=200, sequenceid=784, filesize=14.3 K 2023-04-19 20:55:54,067 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/B/619ac4ba267946ecbd77864f8739f7ab as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/619ac4ba267946ecbd77864f8739f7ab 2023-04-19 20:55:54,070 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/619ac4ba267946ecbd77864f8739f7ab, entries=150, sequenceid=784, filesize=11.9 K 2023-04-19 20:55:54,071 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/C/8b0eefd0196a45b8b810ca16eee6bb25 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/8b0eefd0196a45b8b810ca16eee6bb25 2023-04-19 20:55:54,074 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/8b0eefd0196a45b8b810ca16eee6bb25, entries=150, sequenceid=784, filesize=11.9 K 2023-04-19 20:55:54,074 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 4393734ae917e967cf2b5b4b3b44ed0d in 1248ms, sequenceid=784, compaction requested=true 2023-04-19 20:55:54,074 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 4393734ae917e967cf2b5b4b3b44ed0d: 2023-04-19 20:55:54,075 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 20:55:54,075 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-19 20:55:54,075 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-19 20:55:54,075 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 20:55:54,075 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 20:55:54,075 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 42494 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-19 20:55:54,075 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37614 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-19 20:55:54,075 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1912): 4393734ae917e967cf2b5b4b3b44ed0d/A is initiating minor compaction (all files) 2023-04-19 20:55:54,076 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1912): 4393734ae917e967cf2b5b4b3b44ed0d/B is initiating minor compaction (all files) 2023-04-19 20:55:54,076 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 4393734ae917e967cf2b5b4b3b44ed0d/A in TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:54,076 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4393734ae917e967cf2b5b4b3b44ed0d/B in TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:54,076 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/95eee51867de4585b5da7fe182f100c3, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/3a6b8198c56047fa9f55bce050f5eb83, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/d12bcdc968304759b8bd4a6baa702ec0] into tmpdir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp, totalSize=41.5 K 2023-04-19 20:55:54,076 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/8550f0c81b7046baacb90f80db34e2f3, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/635db6a1d809458d80e73e3b897cf07b, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/619ac4ba267946ecbd77864f8739f7ab] into tmpdir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp, totalSize=36.7 K 2023-04-19 20:55:54,076 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 95eee51867de4585b5da7fe182f100c3, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=732, earliestPutTs=1681937751526 2023-04-19 20:55:54,076 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 8550f0c81b7046baacb90f80db34e2f3, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=732, earliestPutTs=1681937751526 2023-04-19 20:55:54,076 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 3a6b8198c56047fa9f55bce050f5eb83, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=772, earliestPutTs=1681937752375 2023-04-19 20:55:54,076 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 635db6a1d809458d80e73e3b897cf07b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=772, earliestPutTs=1681937752375 2023-04-19 20:55:54,077 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting d12bcdc968304759b8bd4a6baa702ec0, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=784, earliestPutTs=1681937752711 2023-04-19 20:55:54,077 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 619ac4ba267946ecbd77864f8739f7ab, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=784, earliestPutTs=1681937752711 2023-04-19 20:55:54,082 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] throttle.PressureAwareThroughputController(145): 4393734ae917e967cf2b5b4b3b44ed0d#A#compaction#484 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-19 20:55:54,083 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] throttle.PressureAwareThroughputController(145): 4393734ae917e967cf2b5b4b3b44ed0d#B#compaction#485 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 20:55:54,093 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/A/7257ed6fb8374f3ab789ea9c21512964 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/7257ed6fb8374f3ab789ea9c21512964 2023-04-19 20:55:54,097 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/B/824013430a36411f82d84cc69eb916ac as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/824013430a36411f82d84cc69eb916ac 2023-04-19 20:55:54,099 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 4393734ae917e967cf2b5b4b3b44ed0d/A of 4393734ae917e967cf2b5b4b3b44ed0d into 7257ed6fb8374f3ab789ea9c21512964(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 20:55:54,099 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4393734ae917e967cf2b5b4b3b44ed0d: 2023-04-19 20:55:54,099 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d., storeName=4393734ae917e967cf2b5b4b3b44ed0d/A, priority=13, startTime=1681937754075; duration=0sec 2023-04-19 20:55:54,099 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 20:55:54,099 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-19 20:55:54,100 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47288 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-19 20:55:54,100 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1912): 4393734ae917e967cf2b5b4b3b44ed0d/C is initiating minor compaction (all files) 2023-04-19 20:55:54,100 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 4393734ae917e967cf2b5b4b3b44ed0d/C in TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:54,101 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/e7c8e6b0fa934031a1395dcdc0c697e6, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/3e96a02b88904cb09ee3b3488a81a159, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/fedbba22989346bd9f9d053a55b21d12, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/8b0eefd0196a45b8b810ca16eee6bb25] into tmpdir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp, totalSize=46.2 K 2023-04-19 20:55:54,101 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting e7c8e6b0fa934031a1395dcdc0c697e6, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=721, earliestPutTs=1681937751519 2023-04-19 20:55:54,101 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 3e96a02b88904cb09ee3b3488a81a159, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=732, earliestPutTs=1681937752164 2023-04-19 20:55:54,102 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 4393734ae917e967cf2b5b4b3b44ed0d/B of 4393734ae917e967cf2b5b4b3b44ed0d into 824013430a36411f82d84cc69eb916ac(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 20:55:54,102 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting fedbba22989346bd9f9d053a55b21d12, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=772, earliestPutTs=1681937752375 2023-04-19 20:55:54,102 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4393734ae917e967cf2b5b4b3b44ed0d: 2023-04-19 20:55:54,102 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d., storeName=4393734ae917e967cf2b5b4b3b44ed0d/B, priority=13, startTime=1681937754075; duration=0sec 2023-04-19 20:55:54,102 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 20:55:54,102 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 8b0eefd0196a45b8b810ca16eee6bb25, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=784, earliestPutTs=1681937752711 2023-04-19 20:55:54,109 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] throttle.PressureAwareThroughputController(145): 4393734ae917e967cf2b5b4b3b44ed0d#C#compaction#486 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 20:55:54,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(9158): Flush requested on 4393734ae917e967cf2b5b4b3b44ed0d 2023-04-19 20:55:54,111 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 4393734ae917e967cf2b5b4b3b44ed0d 3/3 column families, dataSize=140.89 KB heapSize=400.22 KB 2023-04-19 20:55:54,111 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4393734ae917e967cf2b5b4b3b44ed0d, store=A 2023-04-19 20:55:54,111 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 20:55:54,111 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4393734ae917e967cf2b5b4b3b44ed0d, store=B 2023-04-19 20:55:54,111 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 20:55:54,112 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4393734ae917e967cf2b5b4b3b44ed0d, store=C 2023-04-19 20:55:54,112 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 20:55:54,118 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=824 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/A/67fd81ac447e481bbf4d24eff5e87bda 2023-04-19 20:55:54,120 DEBUG [Thread-3028] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:54,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:54,122 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:54,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:54,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:54,123 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:54,123 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:54,123 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:54,123 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:54,123 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:54,124 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:54,126 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:54,126 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:54,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:54,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:54,126 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:54,126 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:54,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:54,128 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:54,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:54,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:54,129 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:54,129 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:55:54,129 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:54,129 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:54,129 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:54,129 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:54,132 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:54,132 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:54,132 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:54,132 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. started... 2023-04-19 20:55:54,132 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. as already flushing 2023-04-19 20:55:54,132 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:54,132 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:54,132 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:54,132 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:54,132 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:54,132 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:54,134 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:54,134 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:54,134 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:54,135 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:54,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:54,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:54,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:54,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:54,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:54,136 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=824 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/B/e5066d74e0d74d908fa018f82896a5ea 2023-04-19 20:55:54,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:54,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:54,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:54,137 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:54,137 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1f1d59c8[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:54,137 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:54,137 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:54,140 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:54,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:54,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:54,140 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:54,140 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:54,140 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:54,140 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:54,140 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:54,141 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:54,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35486 deadline: 1681937814133, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:54,142 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:54,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:54,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:54,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:54,142 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:54,142 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:54,142 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:54,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:54,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:54,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:54,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:54,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:54,144 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:54,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 277 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35490 deadline: 1681937814136, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:54,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:54,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:54,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:54,146 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:54,146 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:54,146 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:54,146 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:54,146 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:54,146 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:54,147 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:54,147 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:54,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:54,147 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:54,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:54,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:54,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:54,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:54,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:54,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:54,148 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:54,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:54,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:54,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:54,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:54,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:54,149 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:54,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:54,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:54,150 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=824 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/C/d2e9d238194d44daa005f1e915327101 2023-04-19 20:55:54,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:54,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:54,152 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:55:54,152 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:54,152 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:54,152 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:54,152 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:54,152 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:54,152 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:54,152 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:54,152 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:54,152 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:54,152 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:54,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:54,153 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:54,153 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:54,153 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:54,153 DEBUG [Thread-3028] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:54,153 DEBUG [Thread-3028] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:54,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:54,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:54,154 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/A/67fd81ac447e481bbf4d24eff5e87bda as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/67fd81ac447e481bbf4d24eff5e87bda 2023-04-19 20:55:54,157 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/67fd81ac447e481bbf4d24eff5e87bda, entries=250, sequenceid=824, filesize=16.7 K 2023-04-19 20:55:54,157 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/B/e5066d74e0d74d908fa018f82896a5ea as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/e5066d74e0d74d908fa018f82896a5ea 2023-04-19 20:55:54,160 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/e5066d74e0d74d908fa018f82896a5ea, entries=150, sequenceid=824, filesize=11.9 K 2023-04-19 20:55:54,161 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/C/d2e9d238194d44daa005f1e915327101 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/d2e9d238194d44daa005f1e915327101 2023-04-19 20:55:54,164 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/d2e9d238194d44daa005f1e915327101, entries=150, sequenceid=824, filesize=11.9 K 2023-04-19 20:55:54,164 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~400.17 KB/409776, currentSize=40.25 KB/41220 for 4393734ae917e967cf2b5b4b3b44ed0d in 53ms, sequenceid=824, compaction requested=false 2023-04-19 20:55:54,164 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 4393734ae917e967cf2b5b4b3b44ed0d: 2023-04-19 20:55:54,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(9158): Flush requested on 4393734ae917e967cf2b5b4b3b44ed0d 2023-04-19 20:55:54,248 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 4393734ae917e967cf2b5b4b3b44ed0d 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-19 20:55:54,248 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4393734ae917e967cf2b5b4b3b44ed0d, store=A 2023-04-19 20:55:54,248 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:55:54,248 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4393734ae917e967cf2b5b4b3b44ed0d, store=B 2023-04-19 20:55:54,248 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:55:54,248 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4393734ae917e967cf2b5b4b3b44ed0d, store=C 2023-04-19 20:55:54,248 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:55:54,253 DEBUG [Thread-3028] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:54,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:54,255 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:54,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:54,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:54,257 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:54,257 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:54,257 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:54,257 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:54,257 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:54,257 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:54,259 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:54,259 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:54,260 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:54,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:54,260 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:54,260 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:54,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:54,260 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:54,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:54,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:54,261 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:54,261 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:55:54,261 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:54,261 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:54,261 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:54,261 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:54,261 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:54,261 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:54,261 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:54,261 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. started... 2023-04-19 20:55:54,262 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. as already flushing 2023-04-19 20:55:54,262 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:54,262 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:54,262 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:54,262 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:54,262 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:54,262 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:54,263 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:54,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:54,263 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:54,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:54,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:54,264 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:54,264 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:54,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:54,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:54,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:54,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:54,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:54,265 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:54,265 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@fb78711[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:54,265 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:54,265 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:54,266 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:54,266 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:54,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:54,266 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:54,266 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:54,266 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:54,267 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:54,267 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:54,269 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:54,269 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:54,269 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:54,269 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:54,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:54,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:54,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:54,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:54,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:54,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:54,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:54,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:54,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:54,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:54,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:54,272 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:54,272 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:54,272 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:54,272 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:54,272 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:54,272 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:54,275 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=834 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/A/1a5c757d001c44d1bd1cb858c34426e7 2023-04-19 20:55:54,277 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:54,277 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:54,277 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:54,277 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:54,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:54,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:54,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:54,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:54,277 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:54,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:54,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:54,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:54,278 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:54,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:54,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:54,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:54,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:54,282 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:54,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:54,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:54,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:54,285 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:54,285 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:54,285 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:54,285 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:54,285 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:54,285 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:54,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:54,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:54,285 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:54,285 DEBUG [Thread-3028] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:54,285 DEBUG [Thread-3028] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:54,285 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:54,285 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:54,286 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:54,286 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:54,286 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:54,286 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:54,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:54,286 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:55:54,320 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681937754301, max timestamp=1681937754318Num uniques 859; 2023-04-19 20:55:54,320 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681937754301, max timestamp=1681937754318Num uniques 859; 2023-04-19 20:55:54,320 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681937754301, max timestamp=1681937754318Num uniques 859; 2023-04-19 20:55:54,320 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-19 20:55:54,320 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-19 20:55:54,320 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-19 20:55:54,386 DEBUG [Thread-3028] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:54,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:54,388 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:54,389 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:54,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 295 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35490 deadline: 1681937814382, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:54,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:54,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:54,389 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:54,389 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:54,389 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:54,389 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:54,390 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:54,390 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:54,390 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:54,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35486 deadline: 1681937814388, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:54,391 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:54,391 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:54,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:54,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:54,391 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:54,391 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:54,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:54,392 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:54,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:54,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:54,392 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:54,392 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:55:54,393 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:54,393 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:54,393 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:54,393 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:54,393 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:54,393 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:54,393 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:54,393 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. started... 2023-04-19 20:55:54,393 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. as already flushing 2023-04-19 20:55:54,393 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:54,393 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:54,393 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:54,393 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:54,393 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:54,393 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:54,398 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:54,398 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:54,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:54,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:54,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:54,398 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:54,398 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:54,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:54,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:54,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:54,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:54,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:54,399 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:54,400 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1fe4fc73[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:54,400 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:54,400 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:54,402 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:54,402 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:54,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:54,402 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:54,402 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:54,402 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:54,402 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:54,402 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:54,403 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:54,403 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:54,403 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:54,403 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:54,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:54,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:54,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:54,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:54,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:54,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:54,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:54,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:54,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:54,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:54,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:54,406 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:54,406 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:54,406 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:54,406 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:54,406 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:54,406 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:54,408 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:54,408 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:54,408 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:54,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:54,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:54,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:54,408 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:54,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:54,408 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:54,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:54,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:54,409 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:54,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:54,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:54,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:54,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:54,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:54,410 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:54,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:54,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:54,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:54,413 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:54,413 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:54,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:54,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:54,413 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:54,413 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:54,413 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:54,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:54,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:54,413 DEBUG [Thread-3028] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:54,413 DEBUG [Thread-3028] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:54,414 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:54,414 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:54,414 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:54,414 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:54,414 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:54,414 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:54,414 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:54,495 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:54,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 297 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35490 deadline: 1681937814490, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:54,496 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:54,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35486 deadline: 1681937814491, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:54,513 DEBUG [Thread-3028] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:54,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:54,516 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:54,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:54,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:54,517 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:54,517 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:54,517 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:54,517 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:54,518 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:54,518 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:54,518 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/C/50ebf1cf050047ba9d73391a1169e9ee as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/50ebf1cf050047ba9d73391a1169e9ee 2023-04-19 20:55:54,520 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:54,520 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:54,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:54,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:54,521 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:54,521 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:54,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:54,521 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:54,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:54,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:54,522 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:54,522 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:55:54,522 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:54,522 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:54,522 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:54,522 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:54,522 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:54,522 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:54,522 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:54,522 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. started... 2023-04-19 20:55:54,523 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. as already flushing 2023-04-19 20:55:54,523 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:54,523 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 4393734ae917e967cf2b5b4b3b44ed0d/C of 4393734ae917e967cf2b5b4b3b44ed0d into 50ebf1cf050047ba9d73391a1169e9ee(size=13.0 K), total size for store is 24.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 20:55:54,523 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4393734ae917e967cf2b5b4b3b44ed0d: 2023-04-19 20:55:54,523 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d., storeName=4393734ae917e967cf2b5b4b3b44ed0d/C, priority=12, startTime=1681937754075; duration=0sec 2023-04-19 20:55:54,523 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 20:55:54,523 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:54,523 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:54,523 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:54,523 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:54,523 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:54,524 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:54,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:54,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:54,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:54,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:54,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:54,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:54,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:54,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:54,526 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:54,526 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:54,526 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2dceb05b[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:54,526 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:54,526 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:54,526 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:54,526 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:54,527 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:54,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:54,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:54,527 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:54,527 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:54,527 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:54,528 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:54,528 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:54,528 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:54,529 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:54,529 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:54,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:54,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:54,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:54,529 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:54,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:54,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:54,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:54,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:54,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:54,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:54,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:54,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:54,531 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:54,531 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:54,531 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:54,531 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:54,531 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:54,531 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:54,534 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:54,534 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:54,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:54,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:54,534 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:54,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:54,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:54,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:54,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:54,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:54,535 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:54,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:54,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:54,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:54,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:54,536 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:54,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:54,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:54,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:54,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:54,538 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:54,538 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:54,538 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:54,538 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:54,538 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:54,538 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:54,539 DEBUG [Thread-3028] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:54,539 DEBUG [Thread-3028] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:54,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:54,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:54,539 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:54,539 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:54,539 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:54,539 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:54,539 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:54,539 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:54,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:54,539 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:55:54,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:54,639 DEBUG [Thread-3028] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:54,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:54,641 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:54,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:54,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:54,642 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:54,642 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:54,642 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:54,642 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:54,643 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:54,643 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:54,644 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:54,644 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:54,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:54,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:54,645 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:54,645 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:54,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:54,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:54,645 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:54,645 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:54,645 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:54,646 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:55:54,646 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:54,646 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:54,646 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:54,646 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:54,646 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:54,646 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:54,646 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:54,646 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. started... 2023-04-19 20:55:54,646 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. as already flushing 2023-04-19 20:55:54,646 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:54,646 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:54,647 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:54,647 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:54,647 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:54,647 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:54,648 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:54,648 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:54,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:54,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:54,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:54,648 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:54,648 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:54,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:54,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:54,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:54,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:54,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:54,650 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:54,650 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@58e87f33[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:54,650 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:54,650 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:54,652 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:54,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:54,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:54,652 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:54,652 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:54,652 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:54,652 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:54,653 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:54,654 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:54,654 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:54,654 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:54,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:54,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:54,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:54,654 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:54,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:54,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:54,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:54,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:54,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:54,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:54,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:54,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:54,656 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:54,656 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:54,656 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:54,656 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:54,656 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:54,656 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:54,658 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:54,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:54,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:54,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:54,658 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:54,658 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:54,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:54,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:54,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:54,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:54,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:54,659 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:54,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:54,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:54,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:54,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:54,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:54,660 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:54,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:54,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:54,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:54,663 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:54,663 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:54,663 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:54,663 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:54,663 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:55:54,663 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:54,663 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:54,663 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:54,663 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:54,663 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:54,663 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:54,663 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:54,663 DEBUG [Thread-3028] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:54,663 DEBUG [Thread-3028] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:54,663 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:54,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:54,663 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:54,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:54,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:54,695 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=834 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/B/51eddd72661142fb8a4cc4f7e4ba2ffe 2023-04-19 20:55:54,702 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:54,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 299 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35490 deadline: 1681937814696, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:54,702 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:54,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35486 deadline: 1681937814698, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:54,707 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=834 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/C/069b5cab189647e68c3689b0b13ac09b 2023-04-19 20:55:54,716 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/A/1a5c757d001c44d1bd1cb858c34426e7 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/1a5c757d001c44d1bd1cb858c34426e7 2023-04-19 20:55:54,720 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/1a5c757d001c44d1bd1cb858c34426e7, entries=150, sequenceid=834, filesize=11.9 K 2023-04-19 20:55:54,721 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/B/51eddd72661142fb8a4cc4f7e4ba2ffe as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/51eddd72661142fb8a4cc4f7e4ba2ffe 2023-04-19 20:55:54,724 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/51eddd72661142fb8a4cc4f7e4ba2ffe, entries=150, sequenceid=834, filesize=11.9 K 2023-04-19 20:55:54,725 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/C/069b5cab189647e68c3689b0b13ac09b as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/069b5cab189647e68c3689b0b13ac09b 2023-04-19 20:55:54,728 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/069b5cab189647e68c3689b0b13ac09b, entries=150, sequenceid=834, filesize=11.9 K 2023-04-19 20:55:54,729 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 4393734ae917e967cf2b5b4b3b44ed0d in 481ms, sequenceid=834, compaction requested=true 2023-04-19 20:55:54,729 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 4393734ae917e967cf2b5b4b3b44ed0d: 2023-04-19 20:55:54,729 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-19 20:55:54,730 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 20:55:54,730 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-19 20:55:54,732 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 20:55:54,732 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 20:55:54,732 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 42596 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-19 20:55:54,732 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1912): 4393734ae917e967cf2b5b4b3b44ed0d/A is initiating minor compaction (all files) 2023-04-19 20:55:54,732 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37716 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-19 20:55:54,732 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 4393734ae917e967cf2b5b4b3b44ed0d/A in TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:54,732 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1912): 4393734ae917e967cf2b5b4b3b44ed0d/B is initiating minor compaction (all files) 2023-04-19 20:55:54,732 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4393734ae917e967cf2b5b4b3b44ed0d/B in TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:54,732 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/7257ed6fb8374f3ab789ea9c21512964, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/67fd81ac447e481bbf4d24eff5e87bda, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/1a5c757d001c44d1bd1cb858c34426e7] into tmpdir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp, totalSize=41.6 K 2023-04-19 20:55:54,732 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/824013430a36411f82d84cc69eb916ac, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/e5066d74e0d74d908fa018f82896a5ea, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/51eddd72661142fb8a4cc4f7e4ba2ffe] into tmpdir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp, totalSize=36.8 K 2023-04-19 20:55:54,733 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 7257ed6fb8374f3ab789ea9c21512964, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=784, earliestPutTs=1681937752711 2023-04-19 20:55:54,733 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 824013430a36411f82d84cc69eb916ac, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=784, earliestPutTs=1681937752711 2023-04-19 20:55:54,733 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 67fd81ac447e481bbf4d24eff5e87bda, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=824, earliestPutTs=1681937752948 2023-04-19 20:55:54,733 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting e5066d74e0d74d908fa018f82896a5ea, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=824, earliestPutTs=1681937752956 2023-04-19 20:55:54,733 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 1a5c757d001c44d1bd1cb858c34426e7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=834, earliestPutTs=1681937754131 2023-04-19 20:55:54,734 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 51eddd72661142fb8a4cc4f7e4ba2ffe, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=834, earliestPutTs=1681937754131 2023-04-19 20:55:54,745 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] throttle.PressureAwareThroughputController(145): 4393734ae917e967cf2b5b4b3b44ed0d#B#compaction#494 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 20:55:54,745 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] throttle.PressureAwareThroughputController(145): 4393734ae917e967cf2b5b4b3b44ed0d#A#compaction#493 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 20:55:54,763 DEBUG [Thread-3028] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:54,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:54,766 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:54,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:54,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:54,766 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:54,766 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:54,767 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:54,767 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:54,767 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:54,767 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:54,769 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:54,769 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:54,769 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:54,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:54,769 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:54,769 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:54,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:54,769 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:54,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:54,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:54,770 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:54,770 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:55:54,770 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:54,770 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:54,770 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:54,770 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:54,772 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:54,772 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:54,772 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:54,772 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. started... 2023-04-19 20:55:54,772 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4393734ae917e967cf2b5b4b3b44ed0d 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-04-19 20:55:54,772 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4393734ae917e967cf2b5b4b3b44ed0d, store=A 2023-04-19 20:55:54,772 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 20:55:54,772 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4393734ae917e967cf2b5b4b3b44ed0d, store=B 2023-04-19 20:55:54,773 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 20:55:54,773 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4393734ae917e967cf2b5b4b3b44ed0d, store=C 2023-04-19 20:55:54,773 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 20:55:54,782 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=871 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/A/a569198ba4d445669ef81b235c14ff57 2023-04-19 20:55:54,793 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=871 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/B/fa8245b01998409cb6472803c40fbdee 2023-04-19 20:55:54,804 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=871 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/C/2c628db85cf04906bbd04c1a65e4b5fc 2023-04-19 20:55:54,807 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/A/a569198ba4d445669ef81b235c14ff57 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/a569198ba4d445669ef81b235c14ff57 2023-04-19 20:55:54,810 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/a569198ba4d445669ef81b235c14ff57, entries=150, sequenceid=871, filesize=11.9 K 2023-04-19 20:55:54,812 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/B/fa8245b01998409cb6472803c40fbdee as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/fa8245b01998409cb6472803c40fbdee 2023-04-19 20:55:54,815 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/fa8245b01998409cb6472803c40fbdee, entries=150, sequenceid=871, filesize=11.9 K 2023-04-19 20:55:54,816 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/C/2c628db85cf04906bbd04c1a65e4b5fc as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/2c628db85cf04906bbd04c1a65e4b5fc 2023-04-19 20:55:54,821 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/2c628db85cf04906bbd04c1a65e4b5fc, entries=150, sequenceid=871, filesize=11.9 K 2023-04-19 20:55:54,822 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=0 B/0 for 4393734ae917e967cf2b5b4b3b44ed0d in 50ms, sequenceid=871, compaction requested=true 2023-04-19 20:55:54,822 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4393734ae917e967cf2b5b4b3b44ed0d: 2023-04-19 20:55:54,822 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:54,822 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:54,822 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:54,822 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:54,822 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:54,822 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:54,824 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:54,824 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:54,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:54,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:54,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:54,824 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:54,824 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:54,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:54,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:54,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:54,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:54,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:54,826 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:54,826 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6b54e430[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:54,826 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:54,826 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:54,829 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:54,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:54,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:54,829 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:54,829 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:54,829 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:54,829 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:54,830 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:54,832 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:54,832 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:54,832 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:54,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:54,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:54,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:54,832 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:54,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:54,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:54,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:54,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:54,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:54,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:54,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:54,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:54,836 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:54,836 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:54,836 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:54,836 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:54,836 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:54,836 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:54,837 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:54,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:54,837 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:54,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:54,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:54,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:54,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:54,838 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:54,838 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:54,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:54,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:54,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:54,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:54,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:54,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:54,839 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:54,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:54,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:54,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:54,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:54,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:54,843 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:54,843 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:54,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:54,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:54,843 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:54,843 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:54,844 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:54,844 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:54,844 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:54,844 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:54,844 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:54,844 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:54,844 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:54,844 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:54,844 DEBUG [Thread-3028] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:54,844 DEBUG [Thread-3028] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:54,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:54,844 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:55:54,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:54,944 DEBUG [Thread-3028] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:54,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:54,946 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:54,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:54,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:54,947 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:54,947 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:54,947 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:54,947 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:54,948 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:54,948 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:54,949 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:54,949 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:54,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:54,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:54,950 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:54,950 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:54,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:54,950 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:54,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:54,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:54,951 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:54,951 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:55:54,951 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:54,951 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:54,951 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:54,951 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:54,952 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:54,952 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:54,952 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:54,952 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. started... 2023-04-19 20:55:54,952 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4393734ae917e967cf2b5b4b3b44ed0d: 2023-04-19 20:55:54,952 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:54,952 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:54,952 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:54,952 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:54,952 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:54,952 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:54,954 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:54,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:54,954 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:54,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:54,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:54,955 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:54,955 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:54,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:54,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:54,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:54,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:54,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:54,956 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:54,956 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@53eb9a4d[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:54,956 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:54,956 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:54,957 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:54,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:54,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:54,958 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:54,958 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:54,958 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:54,958 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:54,958 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:54,959 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:54,959 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:54,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:54,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:54,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:54,959 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:54,959 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:54,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:54,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:54,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:54,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:54,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:54,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:54,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:54,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:54,963 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:54,963 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:54,963 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:54,963 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:54,963 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:54,963 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:54,964 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:54,964 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:54,964 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:54,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:54,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:54,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:54,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:54,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:54,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:54,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:54,965 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:54,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:54,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:54,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:54,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:54,966 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:54,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:54,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:54,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:54,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:54,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:54,970 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:54,970 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:54,970 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:54,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:54,971 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:54,971 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:54,971 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:54,971 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:54,971 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:54,971 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:54,971 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:54,971 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:54,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:54,971 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:55:54,971 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:54,971 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:54,971 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:54,972 DEBUG [Thread-3028] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:54,972 DEBUG [Thread-3028] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:55,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(9158): Flush requested on 4393734ae917e967cf2b5b4b3b44ed0d 2023-04-19 20:55:55,016 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 4393734ae917e967cf2b5b4b3b44ed0d 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-19 20:55:55,016 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4393734ae917e967cf2b5b4b3b44ed0d, store=A 2023-04-19 20:55:55,016 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:55:55,016 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4393734ae917e967cf2b5b4b3b44ed0d, store=B 2023-04-19 20:55:55,016 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:55:55,016 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4393734ae917e967cf2b5b4b3b44ed0d, store=C 2023-04-19 20:55:55,016 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:55:55,022 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=882 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/A/8dde77a6f07e4cd4a749148263e65e67 2023-04-19 20:55:55,031 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=882 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/B/1b2a0ef114584e59811edfe9b80be235 2023-04-19 20:55:55,057 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=882 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/C/08e34fe8e6c84e8ea85edf580f24ddcc 2023-04-19 20:55:55,061 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/A/8dde77a6f07e4cd4a749148263e65e67 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/8dde77a6f07e4cd4a749148263e65e67 2023-04-19 20:55:55,066 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/8dde77a6f07e4cd4a749148263e65e67, entries=250, sequenceid=882, filesize=16.7 K 2023-04-19 20:55:55,067 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/B/1b2a0ef114584e59811edfe9b80be235 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/1b2a0ef114584e59811edfe9b80be235 2023-04-19 20:55:55,071 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/1b2a0ef114584e59811edfe9b80be235, entries=150, sequenceid=882, filesize=11.9 K 2023-04-19 20:55:55,072 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/C/08e34fe8e6c84e8ea85edf580f24ddcc as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/08e34fe8e6c84e8ea85edf580f24ddcc 2023-04-19 20:55:55,072 DEBUG [Thread-3028] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:55,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:55,075 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:55,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:55,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:55,076 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:55,076 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:55,076 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:55,076 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:55,076 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:55,076 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:55,076 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681937755056, max timestamp=1681937755075Num uniques 859; 2023-04-19 20:55:55,076 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-19 20:55:55,077 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681937755056, max timestamp=1681937755075Num uniques 859; 2023-04-19 20:55:55,077 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-19 20:55:55,078 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/08e34fe8e6c84e8ea85edf580f24ddcc, entries=150, sequenceid=882, filesize=11.9 K 2023-04-19 20:55:55,078 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681937755056, max timestamp=1681937755075Num uniques 859; 2023-04-19 20:55:55,078 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-19 20:55:55,078 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:55,078 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:55,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:55,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:55,078 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=40.25 KB/41220 for 4393734ae917e967cf2b5b4b3b44ed0d in 62ms, sequenceid=882, compaction requested=true 2023-04-19 20:55:55,079 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 4393734ae917e967cf2b5b4b3b44ed0d: 2023-04-19 20:55:55,079 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-19 20:55:55,079 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=3), splitQueue=0 2023-04-19 20:55:55,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:55,079 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=4), splitQueue=0 2023-04-19 20:55:55,079 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:55,079 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:55,079 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:55,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:55,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:55,080 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:55,080 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:55:55,080 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:55,080 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:55,080 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:55,080 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:55,080 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:55,080 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:55,080 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:55,080 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. started... 2023-04-19 20:55:55,080 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4393734ae917e967cf2b5b4b3b44ed0d 3/3 column families, dataSize=40.25 KB heapSize=96.70 KB 2023-04-19 20:55:55,080 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4393734ae917e967cf2b5b4b3b44ed0d, store=A 2023-04-19 20:55:55,080 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 20:55:55,080 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4393734ae917e967cf2b5b4b3b44ed0d, store=B 2023-04-19 20:55:55,081 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 20:55:55,081 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4393734ae917e967cf2b5b4b3b44ed0d, store=C 2023-04-19 20:55:55,081 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 20:55:55,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(9158): Flush requested on 4393734ae917e967cf2b5b4b3b44ed0d 2023-04-19 20:55:55,084 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. as already flushing 2023-04-19 20:55:55,088 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=903 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/A/0458b8415c624f80a1a0f4a75e8d66a7 2023-04-19 20:55:55,126 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1681937755084, max timestamp=1681937755124Num uniques 859; 2023-04-19 20:55:55,126 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1681937755084, max timestamp=1681937755124Num uniques 859; 2023-04-19 20:55:55,126 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-19 20:55:55,126 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1681937755084, max timestamp=1681937755124Num uniques 859; 2023-04-19 20:55:55,126 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-19 20:55:55,126 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-19 20:55:55,162 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:55,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 254 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35492 deadline: 1681937815152, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:55,163 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:55,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 317 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35490 deadline: 1681937815153, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:55,163 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:55,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 279 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35474 deadline: 1681937815160, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:55,165 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/A/ca13e72b97cc4e238f1c3aefc01f007f as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/ca13e72b97cc4e238f1c3aefc01f007f 2023-04-19 20:55:55,166 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/B/b5a024111ffd4368a7b632cfbab91ac9 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/b5a024111ffd4368a7b632cfbab91ac9 2023-04-19 20:55:55,170 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 4393734ae917e967cf2b5b4b3b44ed0d/A of 4393734ae917e967cf2b5b4b3b44ed0d into ca13e72b97cc4e238f1c3aefc01f007f(size=13.1 K), total size for store is 41.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 20:55:55,170 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4393734ae917e967cf2b5b4b3b44ed0d: 2023-04-19 20:55:55,170 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d., storeName=4393734ae917e967cf2b5b4b3b44ed0d/A, priority=13, startTime=1681937754729; duration=0sec 2023-04-19 20:55:55,170 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=4), splitQueue=0 2023-04-19 20:55:55,170 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-19 20:55:55,171 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 4393734ae917e967cf2b5b4b3b44ed0d/B of 4393734ae917e967cf2b5b4b3b44ed0d into b5a024111ffd4368a7b632cfbab91ac9(size=13.1 K), total size for store is 36.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 20:55:55,171 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4393734ae917e967cf2b5b4b3b44ed0d: 2023-04-19 20:55:55,171 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d., storeName=4393734ae917e967cf2b5b4b3b44ed0d/B, priority=13, startTime=1681937754730; duration=0sec 2023-04-19 20:55:55,171 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=3), splitQueue=0 2023-04-19 20:55:55,171 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-19 20:55:55,171 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:55,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 233 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35470 deadline: 1681937815161, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:55,172 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 42698 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-19 20:55:55,172 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1912): 4393734ae917e967cf2b5b4b3b44ed0d/A is initiating minor compaction (all files) 2023-04-19 20:55:55,172 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4393734ae917e967cf2b5b4b3b44ed0d/A in TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:55,172 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/ca13e72b97cc4e238f1c3aefc01f007f, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/a569198ba4d445669ef81b235c14ff57, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/8dde77a6f07e4cd4a749148263e65e67] into tmpdir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp, totalSize=41.7 K 2023-04-19 20:55:55,173 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting ca13e72b97cc4e238f1c3aefc01f007f, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=834, earliestPutTs=1681937754131 2023-04-19 20:55:55,173 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 62054 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-19 20:55:55,173 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting a569198ba4d445669ef81b235c14ff57, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=871, earliestPutTs=1681937754362 2023-04-19 20:55:55,173 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1912): 4393734ae917e967cf2b5b4b3b44ed0d/C is initiating minor compaction (all files) 2023-04-19 20:55:55,173 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 4393734ae917e967cf2b5b4b3b44ed0d/C in TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:55,173 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:55,173 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/50ebf1cf050047ba9d73391a1169e9ee, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/d2e9d238194d44daa005f1e915327101, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/069b5cab189647e68c3689b0b13ac09b, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/2c628db85cf04906bbd04c1a65e4b5fc, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/08e34fe8e6c84e8ea85edf580f24ddcc] into tmpdir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp, totalSize=60.6 K 2023-04-19 20:55:55,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35486 deadline: 1681937815162, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:55,173 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 8dde77a6f07e4cd4a749148263e65e67, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=882, earliestPutTs=1681937755006 2023-04-19 20:55:55,174 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 50ebf1cf050047ba9d73391a1169e9ee, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=784, earliestPutTs=1681937752711 2023-04-19 20:55:55,174 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting d2e9d238194d44daa005f1e915327101, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=824, earliestPutTs=1681937752956 2023-04-19 20:55:55,175 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 069b5cab189647e68c3689b0b13ac09b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=834, earliestPutTs=1681937754131 2023-04-19 20:55:55,175 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 2c628db85cf04906bbd04c1a65e4b5fc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=871, earliestPutTs=1681937754362 2023-04-19 20:55:55,177 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 08e34fe8e6c84e8ea85edf580f24ddcc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=882, earliestPutTs=1681937755010 2023-04-19 20:55:55,181 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] throttle.PressureAwareThroughputController(145): 4393734ae917e967cf2b5b4b3b44ed0d#A#compaction#503 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 20:55:55,184 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] throttle.PressureAwareThroughputController(145): 4393734ae917e967cf2b5b4b3b44ed0d#C#compaction#504 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 20:55:55,198 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/A/2a673f25d2624aa09106d31d7696311e as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/2a673f25d2624aa09106d31d7696311e 2023-04-19 20:55:55,199 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/C/5f9200ec194e4b9ea98ee198625feab6 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/5f9200ec194e4b9ea98ee198625feab6 2023-04-19 20:55:55,203 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 4393734ae917e967cf2b5b4b3b44ed0d/A of 4393734ae917e967cf2b5b4b3b44ed0d into 2a673f25d2624aa09106d31d7696311e(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 20:55:55,203 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4393734ae917e967cf2b5b4b3b44ed0d: 2023-04-19 20:55:55,203 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d., storeName=4393734ae917e967cf2b5b4b3b44ed0d/A, priority=13, startTime=1681937755079; duration=0sec 2023-04-19 20:55:55,203 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-19 20:55:55,203 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-19 20:55:55,205 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37818 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-19 20:55:55,205 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1912): 4393734ae917e967cf2b5b4b3b44ed0d/B is initiating minor compaction (all files) 2023-04-19 20:55:55,205 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4393734ae917e967cf2b5b4b3b44ed0d/B in TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:55,205 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/b5a024111ffd4368a7b632cfbab91ac9, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/fa8245b01998409cb6472803c40fbdee, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/1b2a0ef114584e59811edfe9b80be235] into tmpdir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp, totalSize=36.9 K 2023-04-19 20:55:55,205 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting b5a024111ffd4368a7b632cfbab91ac9, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=834, earliestPutTs=1681937754131 2023-04-19 20:55:55,206 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting fa8245b01998409cb6472803c40fbdee, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=871, earliestPutTs=1681937754362 2023-04-19 20:55:55,206 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 1b2a0ef114584e59811edfe9b80be235, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=882, earliestPutTs=1681937755010 2023-04-19 20:55:55,215 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 4393734ae917e967cf2b5b4b3b44ed0d/C of 4393734ae917e967cf2b5b4b3b44ed0d into 5f9200ec194e4b9ea98ee198625feab6(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 20:55:55,215 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] throttle.PressureAwareThroughputController(145): 4393734ae917e967cf2b5b4b3b44ed0d#B#compaction#505 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 20:55:55,215 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4393734ae917e967cf2b5b4b3b44ed0d: 2023-04-19 20:55:55,215 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d., storeName=4393734ae917e967cf2b5b4b3b44ed0d/C, priority=11, startTime=1681937755079; duration=0sec 2023-04-19 20:55:55,215 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 20:55:55,215 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 1 store files, 0 compacting, 1 eligible, 16 blocking 2023-04-19 20:55:55,216 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-19 20:55:55,216 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-19 20:55:55,216 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. because compaction request was cancelled 2023-04-19 20:55:55,269 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:55,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 256 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35492 deadline: 1681937815265, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:55,270 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:55,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 281 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35474 deadline: 1681937815265, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:55,270 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:55,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 319 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35490 deadline: 1681937815266, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:55,276 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:55,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 235 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35470 deadline: 1681937815272, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:55,276 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:55,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35486 deadline: 1681937815274, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:55,474 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:55,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 258 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35492 deadline: 1681937815470, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:55,474 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:55,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 321 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35490 deadline: 1681937815471, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:55,476 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:55,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 283 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35474 deadline: 1681937815472, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:55,481 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:55,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35486 deadline: 1681937815477, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:55,483 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:55,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 237 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35470 deadline: 1681937815478, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:55,500 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=903 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/B/6858cf98df92400db5f9eea4956bd8a7 2023-04-19 20:55:55,517 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=903 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/C/631c453b03984a5181f72a41b46d0fca 2023-04-19 20:55:55,521 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/A/0458b8415c624f80a1a0f4a75e8d66a7 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/0458b8415c624f80a1a0f4a75e8d66a7 2023-04-19 20:55:55,526 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/0458b8415c624f80a1a0f4a75e8d66a7, entries=150, sequenceid=903, filesize=11.9 K 2023-04-19 20:55:55,527 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/B/6858cf98df92400db5f9eea4956bd8a7 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/6858cf98df92400db5f9eea4956bd8a7 2023-04-19 20:55:55,530 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/6858cf98df92400db5f9eea4956bd8a7, entries=150, sequenceid=903, filesize=11.9 K 2023-04-19 20:55:55,532 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/C/631c453b03984a5181f72a41b46d0fca as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/631c453b03984a5181f72a41b46d0fca 2023-04-19 20:55:55,537 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/631c453b03984a5181f72a41b46d0fca, entries=150, sequenceid=903, filesize=11.9 K 2023-04-19 20:55:55,538 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~96.66 KB/98976, currentSize=154.31 KB/158010 for 4393734ae917e967cf2b5b4b3b44ed0d in 458ms, sequenceid=903, compaction requested=false 2023-04-19 20:55:55,538 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4393734ae917e967cf2b5b4b3b44ed0d: 2023-04-19 20:55:55,538 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:55,538 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:55,538 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:55,538 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:55,538 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:55,538 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:55,539 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:55,539 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:55,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:55,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:55,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:55,540 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:55,540 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:55,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:55,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:55,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:55,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:55,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:55,541 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:55,541 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@bffb403[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:55,541 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:55,541 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:55,542 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:55,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:55,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:55,542 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:55,542 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:55,543 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:55,543 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:55,543 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:55,544 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:55,544 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:55,544 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:55,544 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:55,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:55,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:55,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:55,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:55,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:55,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:55,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:55,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:55,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:55,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:55,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:55,547 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:55,547 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:55,547 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:55,547 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:55,547 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:55,547 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:55,548 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:55,548 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:55,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:55,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:55,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:55,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:55,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:55,548 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:55,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:55,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:55,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:55,549 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:55,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:55,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:55,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:55,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:55,550 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:55,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:55,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:55,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:55,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:55,553 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:55,553 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:55,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:55,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:55,553 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:55,553 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:55,553 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:55,553 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:55:55,553 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:55,553 DEBUG [Thread-3028] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:55,553 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:55,553 DEBUG [Thread-3028] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:55,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:55,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:55,553 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:55,553 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:55,553 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:55,553 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:55,553 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:55,623 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/B/4891ddad549e46d5bf87e3bef789d199 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/4891ddad549e46d5bf87e3bef789d199 2023-04-19 20:55:55,627 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 4393734ae917e967cf2b5b4b3b44ed0d/B of 4393734ae917e967cf2b5b4b3b44ed0d into 4891ddad549e46d5bf87e3bef789d199(size=13.2 K), total size for store is 25.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 20:55:55,627 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4393734ae917e967cf2b5b4b3b44ed0d: 2023-04-19 20:55:55,627 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d., storeName=4393734ae917e967cf2b5b4b3b44ed0d/B, priority=13, startTime=1681937755079; duration=0sec 2023-04-19 20:55:55,627 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 20:55:55,653 DEBUG [Thread-3028] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:55,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:55,655 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:55,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:55,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:55,656 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:55,656 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:55,656 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:55,656 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:55,658 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:55,658 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:55,659 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:55,659 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:55,660 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:55,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:55,660 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:55,660 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:55,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:55,662 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:55,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:55,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:55,662 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:55,662 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:55:55,662 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:55,662 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:55,662 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:55,662 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:55,662 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:55,663 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:55,663 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:55,663 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. started... 2023-04-19 20:55:55,663 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4393734ae917e967cf2b5b4b3b44ed0d 3/3 column families, dataSize=154.31 KB heapSize=430.69 KB 2023-04-19 20:55:55,663 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4393734ae917e967cf2b5b4b3b44ed0d, store=A 2023-04-19 20:55:55,663 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 20:55:55,663 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4393734ae917e967cf2b5b4b3b44ed0d, store=B 2023-04-19 20:55:55,663 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 20:55:55,664 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4393734ae917e967cf2b5b4b3b44ed0d, store=C 2023-04-19 20:55:55,664 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 20:55:55,673 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=945 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/A/309858ab27fc48369f1d6a4dd7bd78f6 2023-04-19 20:55:55,683 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=945 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/B/38e1cc4375034e13b2652139e727fdf5 2023-04-19 20:55:55,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(9158): Flush requested on 4393734ae917e967cf2b5b4b3b44ed0d 2023-04-19 20:55:55,783 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. as already flushing 2023-04-19 20:55:55,791 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:55,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 286 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35474 deadline: 1681937815784, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:55,791 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:55,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35470 deadline: 1681937815785, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:55,797 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:55,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 325 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35490 deadline: 1681937815790, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:55,799 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:55,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 262 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35492 deadline: 1681937815791, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:55,800 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:55,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35486 deadline: 1681937815791, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:55,898 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:55,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 288 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35474 deadline: 1681937815892, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:55,905 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:55,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 327 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35490 deadline: 1681937815898, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:55,910 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:55,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 264 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35492 deadline: 1681937815900, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:55,910 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:55,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35486 deadline: 1681937815904, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:56,098 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=945 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/C/ab2fb1ab7b024df587d4d03f94257985 2023-04-19 20:55:56,102 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/A/309858ab27fc48369f1d6a4dd7bd78f6 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/309858ab27fc48369f1d6a4dd7bd78f6 2023-04-19 20:55:56,105 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:56,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 290 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35474 deadline: 1681937816100, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:56,105 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/309858ab27fc48369f1d6a4dd7bd78f6, entries=150, sequenceid=945, filesize=11.9 K 2023-04-19 20:55:56,106 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/B/38e1cc4375034e13b2652139e727fdf5 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/38e1cc4375034e13b2652139e727fdf5 2023-04-19 20:55:56,110 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/38e1cc4375034e13b2652139e727fdf5, entries=150, sequenceid=945, filesize=11.9 K 2023-04-19 20:55:56,110 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/C/ab2fb1ab7b024df587d4d03f94257985 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/ab2fb1ab7b024df587d4d03f94257985 2023-04-19 20:55:56,114 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/ab2fb1ab7b024df587d4d03f94257985, entries=150, sequenceid=945, filesize=11.9 K 2023-04-19 20:55:56,115 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~154.31 KB/158010, heapSize ~430.64 KB/440976, currentSize=40.25 KB/41220 for 4393734ae917e967cf2b5b4b3b44ed0d in 452ms, sequenceid=945, compaction requested=true 2023-04-19 20:55:56,115 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4393734ae917e967cf2b5b4b3b44ed0d: 2023-04-19 20:55:56,115 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:56,115 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:56,115 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:56,115 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:56,115 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:56,115 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:56,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(9158): Flush requested on 4393734ae917e967cf2b5b4b3b44ed0d 2023-04-19 20:55:56,117 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 4393734ae917e967cf2b5b4b3b44ed0d 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-19 20:55:56,117 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4393734ae917e967cf2b5b4b3b44ed0d, store=A 2023-04-19 20:55:56,117 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:55:56,117 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4393734ae917e967cf2b5b4b3b44ed0d, store=B 2023-04-19 20:55:56,117 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:55:56,117 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4393734ae917e967cf2b5b4b3b44ed0d, store=C 2023-04-19 20:55:56,117 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:55:56,118 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:56,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:56,118 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:56,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:56,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:56,119 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:56,119 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:56,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:56,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:56,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:56,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:56,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:56,120 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:56,121 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@107543a8[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:56,121 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:56,121 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:56,122 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:56,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:56,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:56,122 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:56,122 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:56,122 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:56,124 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:56,124 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:56,126 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:56,126 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:56,126 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:56,126 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:56,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:56,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:56,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:56,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:56,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:56,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:56,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:56,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:56,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:56,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:56,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:56,129 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:56,129 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:56,129 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:56,129 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:56,129 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:56,129 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:56,131 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:56,131 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:56,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:56,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:56,131 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:56,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:56,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:56,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:56,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:56,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:56,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:56,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:56,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:56,133 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:56,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:56,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:56,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:56,134 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:56,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:56,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:56,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:56,136 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:56,136 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:56,136 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:56,136 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:56,136 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:56,137 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:56,137 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:56,137 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:56,137 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:56,137 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:56,137 DEBUG [Thread-3028] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:56,137 DEBUG [Thread-3028] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:56,138 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:56,138 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:56,138 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:56,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:56,138 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:56,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:56,194 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:56,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 272 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35492 deadline: 1681937816186, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:56,194 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681937756146, max timestamp=1681937756191Num uniques 859; 2023-04-19 20:55:56,195 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 20:55:56,194 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681937756146, max timestamp=1681937756191Num uniques 859; 2023-04-19 20:55:56,194 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681937756146, max timestamp=1681937756191Num uniques 859; 2023-04-19 20:55:56,195 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 20:55:56,195 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 20:55:56,237 DEBUG [Thread-3028] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:56,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:56,239 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:56,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:56,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:56,240 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:56,240 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:56,240 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:56,240 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:56,240 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:56,240 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:56,241 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:56,241 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:56,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:56,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:56,242 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:56,242 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:56,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:56,242 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:56,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:56,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:56,243 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:56,243 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:55:56,243 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:56,243 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:56,243 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:56,243 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:56,243 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:56,243 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:56,243 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:56,243 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. started... 2023-04-19 20:55:56,243 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. as already flushing 2023-04-19 20:55:56,244 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:56,244 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:56,244 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:56,244 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:56,244 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:56,244 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:56,246 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:56,246 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:56,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:56,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:56,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:56,246 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:56,246 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:56,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:56,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:56,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:56,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:56,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:56,247 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:56,247 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c4c40f2[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:56,247 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:56,247 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:56,250 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:56,250 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:56,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:56,250 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:56,250 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:56,250 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:56,250 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:56,250 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:56,252 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:56,252 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:56,252 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:56,252 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:56,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:56,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:56,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:56,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:56,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:56,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:56,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:56,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:56,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:56,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:56,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:56,254 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:56,254 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:56,254 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:56,254 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:56,254 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:56,254 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:56,256 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:56,256 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:56,256 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:56,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:56,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:56,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:56,256 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:56,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:56,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:56,256 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:56,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:56,256 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:56,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:56,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:56,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:56,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:56,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:56,257 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:56,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:56,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:56,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:56,259 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:56,259 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:56,259 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:56,259 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:56,259 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:56,259 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:56,260 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:56,259 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:56,260 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:56,260 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:56,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:56,260 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:56,260 DEBUG [Thread-3028] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:56,260 DEBUG [Thread-3028] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:56,260 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:56,260 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:56,260 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:56,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:56,293 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1681937756266, max timestamp=1681937756290Num uniques 1018; 2023-04-19 20:55:56,294 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1681937756266, max timestamp=1681937756290Num uniques 1018; 2023-04-19 20:55:56,294 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-19 20:55:56,294 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-19 20:55:56,294 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1681937756266, max timestamp=1681937756290Num uniques 1018; 2023-04-19 20:55:56,294 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-19 20:55:56,342 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:56,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 244 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35470 deadline: 1681937816333, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:56,345 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:56,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 348 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35490 deadline: 1681937816333, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:56,346 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:56,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 237 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35486 deadline: 1681937816335, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:56,349 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:56,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 278 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35492 deadline: 1681937816342, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:56,360 DEBUG [Thread-3028] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:56,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:56,362 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:56,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:56,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:56,363 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:56,364 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:56,364 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:56,364 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:56,364 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:56,364 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:56,366 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:56,366 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:56,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:56,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:56,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:56,367 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:56,367 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:56,367 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:56,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:56,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:56,367 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:56,368 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:55:56,368 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:56,368 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:56,368 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:56,368 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:56,368 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:56,368 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:56,368 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:56,368 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. started... 2023-04-19 20:55:56,368 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. as already flushing 2023-04-19 20:55:56,368 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:56,368 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:56,368 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:56,368 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:56,368 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:56,368 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:56,370 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:56,370 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:56,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:56,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:56,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:56,370 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:56,370 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:56,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:56,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:56,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:56,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:56,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:56,371 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:56,371 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@537a90c4[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:56,371 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:56,371 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:56,373 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:56,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:56,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:56,373 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:56,373 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:56,373 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:56,373 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:56,373 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:56,374 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:56,374 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:56,374 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:56,375 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:56,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:56,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:56,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:56,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:56,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:56,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:56,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:56,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:56,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:56,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:56,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:56,377 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:56,377 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:56,378 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:56,378 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:56,378 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:56,378 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:56,380 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:56,380 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:56,380 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:56,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:56,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:56,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:56,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:56,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:56,381 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:56,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:56,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:56,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:56,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:56,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:56,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:56,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:56,384 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:56,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:56,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:56,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:56,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:56,386 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:56,386 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:56,386 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:56,386 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:56,386 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:56,386 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:56,386 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:56,386 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:56,386 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:56,386 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:56,386 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:56,386 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:56,386 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:56,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:56,386 DEBUG [Thread-3028] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:56,386 DEBUG [Thread-3028] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:56,387 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:55:56,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:56,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:56,418 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:56,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 292 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35474 deadline: 1681937816408, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:56,446 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:56,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 246 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35470 deadline: 1681937816443, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:56,456 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:56,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 350 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35490 deadline: 1681937816447, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:56,456 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:56,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35486 deadline: 1681937816447, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:56,458 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:56,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 280 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35492 deadline: 1681937816450, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:56,487 DEBUG [Thread-3028] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:56,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:56,489 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:56,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:56,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:56,490 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:56,490 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:56,490 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:56,490 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:56,491 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:56,491 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:56,492 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:56,492 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:56,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:56,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:56,492 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:56,492 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:56,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:56,493 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:56,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:56,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:56,493 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:56,493 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:55:56,494 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:56,494 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:56,494 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:56,494 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:56,494 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:56,494 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:56,494 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:56,494 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. started... 2023-04-19 20:55:56,494 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. as already flushing 2023-04-19 20:55:56,494 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:56,494 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:56,494 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:56,494 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:56,494 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:56,494 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:56,495 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:56,496 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:56,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:56,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:56,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:56,496 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:56,496 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:56,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:56,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:56,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:56,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:56,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:56,497 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:56,497 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@25d37904[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:56,497 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:56,497 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:56,500 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:56,500 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:56,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:56,500 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:56,500 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:56,500 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:56,500 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:56,500 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:56,501 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:56,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:56,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:56,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:56,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:56,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:56,502 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:56,502 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:56,502 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:56,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:56,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:56,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:56,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:56,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:56,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:56,503 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:56,503 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:56,503 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:56,503 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:56,503 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:56,504 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:56,505 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:56,505 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:56,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:56,505 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:56,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:56,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:56,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:56,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:56,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:56,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:56,505 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:56,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:56,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:56,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:56,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:56,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:56,507 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:56,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:56,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:56,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:56,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:56,513 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:56,513 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:56,513 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:56,513 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:56,513 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:56,513 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:56,514 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:56,513 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:56,514 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:56,514 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:56,514 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:56,514 DEBUG [Thread-3028] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:56,514 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:56,514 DEBUG [Thread-3028] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:56,514 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:56,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:56,514 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:56,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:56,533 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=955 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/A/e4ea900cb7b249c690f5dabded681cf9 2023-04-19 20:55:56,550 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=955 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/B/2d5bfb8ba7604e86a0ee3f058a27f2ca 2023-04-19 20:55:56,559 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=955 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/C/1864efc9f6194856b7fcf77ec52cbab8 2023-04-19 20:55:56,562 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/A/e4ea900cb7b249c690f5dabded681cf9 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/e4ea900cb7b249c690f5dabded681cf9 2023-04-19 20:55:56,564 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/e4ea900cb7b249c690f5dabded681cf9, entries=200, sequenceid=955, filesize=14.3 K 2023-04-19 20:55:56,565 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/B/2d5bfb8ba7604e86a0ee3f058a27f2ca as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/2d5bfb8ba7604e86a0ee3f058a27f2ca 2023-04-19 20:55:56,569 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/2d5bfb8ba7604e86a0ee3f058a27f2ca, entries=150, sequenceid=955, filesize=11.9 K 2023-04-19 20:55:56,570 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/C/1864efc9f6194856b7fcf77ec52cbab8 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/1864efc9f6194856b7fcf77ec52cbab8 2023-04-19 20:55:56,572 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/1864efc9f6194856b7fcf77ec52cbab8, entries=150, sequenceid=955, filesize=11.9 K 2023-04-19 20:55:56,573 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 4393734ae917e967cf2b5b4b3b44ed0d in 456ms, sequenceid=955, compaction requested=true 2023-04-19 20:55:56,573 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 4393734ae917e967cf2b5b4b3b44ed0d: 2023-04-19 20:55:56,573 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 20:55:56,573 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-19 20:55:56,573 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 20:55:56,573 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-19 20:55:56,573 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 20:55:56,574 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 52546 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-19 20:55:56,574 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1912): 4393734ae917e967cf2b5b4b3b44ed0d/A is initiating minor compaction (all files) 2023-04-19 20:55:56,574 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 50106 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-19 20:55:56,575 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 4393734ae917e967cf2b5b4b3b44ed0d/A in TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:56,575 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1912): 4393734ae917e967cf2b5b4b3b44ed0d/B is initiating minor compaction (all files) 2023-04-19 20:55:56,575 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/2a673f25d2624aa09106d31d7696311e, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/0458b8415c624f80a1a0f4a75e8d66a7, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/309858ab27fc48369f1d6a4dd7bd78f6, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/e4ea900cb7b249c690f5dabded681cf9] into tmpdir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp, totalSize=51.3 K 2023-04-19 20:55:56,575 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4393734ae917e967cf2b5b4b3b44ed0d/B in TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:56,575 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/4891ddad549e46d5bf87e3bef789d199, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/6858cf98df92400db5f9eea4956bd8a7, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/38e1cc4375034e13b2652139e727fdf5, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/2d5bfb8ba7604e86a0ee3f058a27f2ca] into tmpdir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp, totalSize=48.9 K 2023-04-19 20:55:56,575 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 2a673f25d2624aa09106d31d7696311e, keycount=150, bloomtype=ROW, size=13.2 K, encoding=NONE, compression=NONE, seqNum=882, earliestPutTs=1681937755010 2023-04-19 20:55:56,576 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 4891ddad549e46d5bf87e3bef789d199, keycount=150, bloomtype=ROW, size=13.2 K, encoding=NONE, compression=NONE, seqNum=882, earliestPutTs=1681937755010 2023-04-19 20:55:56,576 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 0458b8415c624f80a1a0f4a75e8d66a7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=903, earliestPutTs=1681937755066 2023-04-19 20:55:56,577 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 6858cf98df92400db5f9eea4956bd8a7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=903, earliestPutTs=1681937755066 2023-04-19 20:55:56,577 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 309858ab27fc48369f1d6a4dd7bd78f6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=945, earliestPutTs=1681937755127 2023-04-19 20:55:56,577 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 38e1cc4375034e13b2652139e727fdf5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=945, earliestPutTs=1681937755127 2023-04-19 20:55:56,577 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting e4ea900cb7b249c690f5dabded681cf9, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=955, earliestPutTs=1681937755789 2023-04-19 20:55:56,578 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 2d5bfb8ba7604e86a0ee3f058a27f2ca, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=955, earliestPutTs=1681937755790 2023-04-19 20:55:56,583 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] throttle.PressureAwareThroughputController(145): 4393734ae917e967cf2b5b4b3b44ed0d#A#compaction#513 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 20:55:56,585 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] throttle.PressureAwareThroughputController(145): 4393734ae917e967cf2b5b4b3b44ed0d#B#compaction#514 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 20:55:56,603 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/B/f300d42c7c5048619332277a7d304dae as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/f300d42c7c5048619332277a7d304dae 2023-04-19 20:55:56,606 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 4393734ae917e967cf2b5b4b3b44ed0d/B of 4393734ae917e967cf2b5b4b3b44ed0d into f300d42c7c5048619332277a7d304dae(size=13.4 K), total size for store is 13.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 20:55:56,606 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4393734ae917e967cf2b5b4b3b44ed0d: 2023-04-19 20:55:56,606 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d., storeName=4393734ae917e967cf2b5b4b3b44ed0d/B, priority=12, startTime=1681937756573; duration=0sec 2023-04-19 20:55:56,607 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 20:55:56,607 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-19 20:55:56,608 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 50038 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-19 20:55:56,608 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1912): 4393734ae917e967cf2b5b4b3b44ed0d/C is initiating minor compaction (all files) 2023-04-19 20:55:56,608 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4393734ae917e967cf2b5b4b3b44ed0d/C in TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:56,608 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/5f9200ec194e4b9ea98ee198625feab6, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/631c453b03984a5181f72a41b46d0fca, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/ab2fb1ab7b024df587d4d03f94257985, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/1864efc9f6194856b7fcf77ec52cbab8] into tmpdir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp, totalSize=48.9 K 2023-04-19 20:55:56,608 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 5f9200ec194e4b9ea98ee198625feab6, keycount=150, bloomtype=ROW, size=13.2 K, encoding=NONE, compression=NONE, seqNum=882, earliestPutTs=1681937755010 2023-04-19 20:55:56,608 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 631c453b03984a5181f72a41b46d0fca, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=903, earliestPutTs=1681937755066 2023-04-19 20:55:56,609 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting ab2fb1ab7b024df587d4d03f94257985, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=945, earliestPutTs=1681937755127 2023-04-19 20:55:56,609 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 1864efc9f6194856b7fcf77ec52cbab8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=955, earliestPutTs=1681937755790 2023-04-19 20:55:56,614 DEBUG [Thread-3028] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:56,614 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] throttle.PressureAwareThroughputController(145): 4393734ae917e967cf2b5b4b3b44ed0d#C#compaction#515 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 20:55:56,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:56,616 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:56,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:56,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:56,627 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:56,627 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:56,627 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:56,627 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:56,627 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:56,627 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:56,629 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:56,629 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:56,629 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:56,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:56,629 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:56,629 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:56,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:56,629 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:56,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:56,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:56,630 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:56,630 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:55:56,630 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:56,630 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:56,630 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:56,630 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:56,630 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/C/b634153cab4148388e9e607cbbdcdf58 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/b634153cab4148388e9e607cbbdcdf58 2023-04-19 20:55:56,630 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:56,630 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:56,630 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. started... 2023-04-19 20:55:56,630 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:56,630 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4393734ae917e967cf2b5b4b3b44ed0d 3/3 column families, dataSize=134.18 KB heapSize=379.45 KB 2023-04-19 20:55:56,631 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4393734ae917e967cf2b5b4b3b44ed0d, store=A 2023-04-19 20:55:56,631 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 20:55:56,631 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4393734ae917e967cf2b5b4b3b44ed0d, store=B 2023-04-19 20:55:56,631 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 20:55:56,631 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4393734ae917e967cf2b5b4b3b44ed0d, store=C 2023-04-19 20:55:56,631 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 20:55:56,635 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 4393734ae917e967cf2b5b4b3b44ed0d/C of 4393734ae917e967cf2b5b4b3b44ed0d into b634153cab4148388e9e607cbbdcdf58(size=13.3 K), total size for store is 13.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 20:55:56,635 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4393734ae917e967cf2b5b4b3b44ed0d: 2023-04-19 20:55:56,635 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d., storeName=4393734ae917e967cf2b5b4b3b44ed0d/C, priority=12, startTime=1681937756573; duration=0sec 2023-04-19 20:55:56,635 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 20:55:56,643 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=1009 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/A/fd79b6156cd94489820ffcc7eada3452 2023-04-19 20:55:56,654 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=1009 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/B/b2cf2936dbe54253b08855f45730c8be 2023-04-19 20:55:56,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(9158): Flush requested on 4393734ae917e967cf2b5b4b3b44ed0d 2023-04-19 20:55:56,655 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. as already flushing 2023-04-19 20:55:56,674 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:56,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 251 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35470 deadline: 1681937816667, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:56,675 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:56,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 283 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35492 deadline: 1681937816668, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:56,678 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:56,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 354 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35490 deadline: 1681937816674, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:56,680 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:56,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 243 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35486 deadline: 1681937816675, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:56,780 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:56,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 285 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35492 deadline: 1681937816776, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:56,781 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:56,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 253 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35470 deadline: 1681937816776, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:56,788 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:56,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 356 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35490 deadline: 1681937816780, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:56,789 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:56,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 245 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35486 deadline: 1681937816781, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:56,926 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:56,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 294 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35474 deadline: 1681937816922, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:56,986 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:56,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 287 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35492 deadline: 1681937816983, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:56,987 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:56,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 255 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35470 deadline: 1681937816983, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:56,994 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:56,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 247 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35486 deadline: 1681937816990, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:56,995 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:56,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 358 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35490 deadline: 1681937816991, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:56,997 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/A/548fac84597544b7aa3353b95607a054 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/548fac84597544b7aa3353b95607a054 2023-04-19 20:55:57,001 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 4393734ae917e967cf2b5b4b3b44ed0d/A of 4393734ae917e967cf2b5b4b3b44ed0d into 548fac84597544b7aa3353b95607a054(size=13.4 K), total size for store is 13.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 20:55:57,001 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4393734ae917e967cf2b5b4b3b44ed0d: 2023-04-19 20:55:57,001 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d., storeName=4393734ae917e967cf2b5b4b3b44ed0d/A, priority=12, startTime=1681937756573; duration=0sec 2023-04-19 20:55:57,002 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 20:55:57,070 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=1009 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/C/5397c2fbad0249d4a559380e55097d25 2023-04-19 20:55:57,073 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/A/fd79b6156cd94489820ffcc7eada3452 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/fd79b6156cd94489820ffcc7eada3452 2023-04-19 20:55:57,076 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/fd79b6156cd94489820ffcc7eada3452, entries=150, sequenceid=1009, filesize=11.9 K 2023-04-19 20:55:57,077 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/B/b2cf2936dbe54253b08855f45730c8be as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/b2cf2936dbe54253b08855f45730c8be 2023-04-19 20:55:57,080 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/b2cf2936dbe54253b08855f45730c8be, entries=150, sequenceid=1009, filesize=11.9 K 2023-04-19 20:55:57,080 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/C/5397c2fbad0249d4a559380e55097d25 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/5397c2fbad0249d4a559380e55097d25 2023-04-19 20:55:57,083 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/5397c2fbad0249d4a559380e55097d25, entries=150, sequenceid=1009, filesize=11.9 K 2023-04-19 20:55:57,084 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~379.41 KB/388512, currentSize=53.67 KB/54960 for 4393734ae917e967cf2b5b4b3b44ed0d in 454ms, sequenceid=1009, compaction requested=false 2023-04-19 20:55:57,084 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4393734ae917e967cf2b5b4b3b44ed0d: 2023-04-19 20:55:57,084 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:57,084 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:57,084 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:57,084 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:57,084 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:57,084 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:57,086 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:57,086 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:57,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:57,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:57,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:57,086 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:57,086 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:57,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:57,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:57,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:57,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:57,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:57,088 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:57,088 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1f811615[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:57,088 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:57,088 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:57,089 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:57,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:57,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:57,089 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:57,089 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:57,089 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:57,089 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:57,089 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:57,091 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:57,091 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:57,091 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:57,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:57,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:57,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:57,091 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:57,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:57,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:57,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:57,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:57,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:57,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:57,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:57,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:57,094 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:57,094 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:57,094 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:57,094 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:57,094 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:57,094 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:57,096 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:57,096 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:57,096 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:57,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:57,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:57,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:57,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:57,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:57,097 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:57,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:57,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:57,097 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:57,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:57,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:57,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:57,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:57,098 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:57,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:57,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:57,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:57,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:57,101 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:57,101 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:57,101 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:57,101 DEBUG [Thread-3028] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:57,101 DEBUG [Thread-3028] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:57,101 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:55:57,101 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:57,102 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:57,102 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:57,102 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:57,102 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:57,102 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:57,102 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:57,102 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:57,102 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:57,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:57,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:57,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:57,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:57,201 DEBUG [Thread-3028] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:57,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:57,203 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:57,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:57,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:57,205 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:57,205 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:57,205 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:57,205 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:57,205 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:57,205 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:57,206 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:57,206 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:57,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:57,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:57,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:57,207 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:57,207 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:57,207 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:57,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:57,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:57,208 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:57,208 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:55:57,208 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:57,208 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:57,208 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:57,208 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:57,208 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:57,208 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:57,209 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:57,209 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. started... 2023-04-19 20:55:57,209 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4393734ae917e967cf2b5b4b3b44ed0d 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-19 20:55:57,209 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4393734ae917e967cf2b5b4b3b44ed0d, store=A 2023-04-19 20:55:57,209 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:55:57,209 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4393734ae917e967cf2b5b4b3b44ed0d, store=B 2023-04-19 20:55:57,209 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:55:57,209 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4393734ae917e967cf2b5b4b3b44ed0d, store=C 2023-04-19 20:55:57,209 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:55:57,216 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=1022 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/A/318be65f80634fa0b4defcc935446002 2023-04-19 20:55:57,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(9158): Flush requested on 4393734ae917e967cf2b5b4b3b44ed0d 2023-04-19 20:55:57,291 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. as already flushing 2023-04-19 20:55:57,326 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1681937757296, max timestamp=1681937757324Num uniques 859; 2023-04-19 20:55:57,326 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1681937757296, max timestamp=1681937757324Num uniques 859; 2023-04-19 20:55:57,326 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-19 20:55:57,326 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-19 20:55:57,326 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1681937757296, max timestamp=1681937757324Num uniques 859; 2023-04-19 20:55:57,326 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-19 20:55:57,358 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:57,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 366 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35490 deadline: 1681937817352, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:57,360 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:57,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 266 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35470 deadline: 1681937817353, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:57,361 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:57,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 256 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35486 deadline: 1681937817354, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:57,363 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:57,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 298 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35492 deadline: 1681937817355, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:57,462 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:57,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 368 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35490 deadline: 1681937817459, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:57,470 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:57,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 268 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35470 deadline: 1681937817461, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:57,470 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:57,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 258 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35486 deadline: 1681937817461, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:57,471 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:57,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 300 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35492 deadline: 1681937817464, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:57,629 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=1022 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/B/7b941f46c179455e9ed81fe241ff0761 2023-04-19 20:55:57,640 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=1022 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/C/ff31eae303cc4f7f83a58ec6b50c69ea 2023-04-19 20:55:57,644 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/A/318be65f80634fa0b4defcc935446002 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/318be65f80634fa0b4defcc935446002 2023-04-19 20:55:57,654 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/318be65f80634fa0b4defcc935446002, entries=150, sequenceid=1022, filesize=11.9 K 2023-04-19 20:55:57,655 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/B/7b941f46c179455e9ed81fe241ff0761 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/7b941f46c179455e9ed81fe241ff0761 2023-04-19 20:55:57,659 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/7b941f46c179455e9ed81fe241ff0761, entries=150, sequenceid=1022, filesize=11.9 K 2023-04-19 20:55:57,660 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/C/ff31eae303cc4f7f83a58ec6b50c69ea as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/ff31eae303cc4f7f83a58ec6b50c69ea 2023-04-19 20:55:57,670 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:57,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 370 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35490 deadline: 1681937817667, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:57,673 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/ff31eae303cc4f7f83a58ec6b50c69ea, entries=150, sequenceid=1022, filesize=11.9 K 2023-04-19 20:55:57,674 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=134.18 KB/137400 for 4393734ae917e967cf2b5b4b3b44ed0d in 465ms, sequenceid=1022, compaction requested=true 2023-04-19 20:55:57,674 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4393734ae917e967cf2b5b4b3b44ed0d: 2023-04-19 20:55:57,674 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:57,674 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:57,674 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:57,674 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:57,674 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:57,675 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:57,677 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:57,677 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:57,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:57,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:57,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:57,677 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:57,677 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:57,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:57,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(9158): Flush requested on 4393734ae917e967cf2b5b4b3b44ed0d 2023-04-19 20:55:57,678 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 4393734ae917e967cf2b5b4b3b44ed0d 3/3 column families, dataSize=140.89 KB heapSize=390.84 KB 2023-04-19 20:55:57,678 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4393734ae917e967cf2b5b4b3b44ed0d, store=A 2023-04-19 20:55:57,678 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 20:55:57,678 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4393734ae917e967cf2b5b4b3b44ed0d, store=B 2023-04-19 20:55:57,679 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 20:55:57,679 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4393734ae917e967cf2b5b4b3b44ed0d, store=C 2023-04-19 20:55:57,679 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 20:55:57,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:57,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:57,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:57,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:57,680 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:57,680 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@21748ec7[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:57,680 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:57,680 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:57,682 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:57,682 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:57,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:57,682 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:57,682 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:57,682 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:57,682 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:57,682 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:57,683 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:57,683 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:57,683 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:57,685 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:57,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:57,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:57,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:57,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:57,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:57,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:57,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:57,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:57,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:57,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:57,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:57,688 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:57,688 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:57,688 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:57,688 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:57,688 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:57,688 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:57,690 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:57,690 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:57,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:57,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:57,690 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:57,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:57,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:57,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:57,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:57,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:57,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:57,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:57,691 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:57,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:57,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:57,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:57,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:57,693 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:57,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:57,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:57,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:57,696 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:57,696 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:57,696 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:57,696 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:57,696 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:57,696 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:57,696 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:57,697 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:57,697 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:57,697 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:57,697 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:57,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:57,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:57,697 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:57,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:57,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:57,698 DEBUG [Thread-3028] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:57,698 DEBUG [Thread-3028] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:57,706 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:57,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 272 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35470 deadline: 1681937817698, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:57,708 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:57,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 263 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35486 deadline: 1681937817699, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:57,713 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:57,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 305 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35492 deadline: 1681937817706, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:57,798 DEBUG [Thread-3028] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:57,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:57,800 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:57,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:57,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:57,801 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:57,801 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:57,801 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:57,801 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:57,804 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:57,804 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:57,805 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:57,805 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:57,805 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:57,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:57,806 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:57,806 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:57,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:57,806 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:57,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:57,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:57,807 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:57,807 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:55:57,807 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:57,807 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:57,807 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:57,807 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:57,807 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:57,807 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:57,807 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:57,807 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. started... 2023-04-19 20:55:57,807 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. as already flushing 2023-04-19 20:55:57,807 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:57,807 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:57,807 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:57,807 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:57,808 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:57,808 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:57,809 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:57,809 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:57,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:57,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:57,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:57,810 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:57,810 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:57,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:57,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:57,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:57,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:57,811 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:57,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:57,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 274 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35470 deadline: 1681937817807, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:57,812 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:57,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 265 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35486 deadline: 1681937817809, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:57,812 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:57,812 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@268fd69d[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:57,812 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:57,812 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:57,813 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:57,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:57,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:57,813 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:57,813 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:57,813 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:57,813 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:57,814 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:57,816 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:57,816 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:57,816 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:57,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:57,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:57,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:57,816 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:57,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:57,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:57,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:57,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:57,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:57,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:57,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:57,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:57,818 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:57,818 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:57,818 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:57,818 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:57,818 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:57,818 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:57,819 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:57,819 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:57,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:57,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:57,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:57,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:57,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:57,820 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:57,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:57,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:57,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:57,821 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:57,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 307 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35492 deadline: 1681937817814, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:57,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:57,821 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:57,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:57,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:57,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:57,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:57,822 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:57,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:57,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:57,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:57,826 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:57,826 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:57,826 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:57,826 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:57,827 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:57,826 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:57,826 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:57,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:57,826 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:57,827 DEBUG [Thread-3028] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:57,827 DEBUG [Thread-3028] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:57,827 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:57,827 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:57,827 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:57,827 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:57,827 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:57,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:57,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:57,927 DEBUG [Thread-3028] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:57,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:57,929 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:57,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:57,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:57,930 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:57,930 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:57,930 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:57,930 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:57,930 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:57,930 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:57,932 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:57,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:57,932 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:57,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:57,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:57,933 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:57,933 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:57,933 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:57,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:57,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:57,933 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:57,933 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:55:57,933 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:57,933 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:57,934 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:57,934 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:57,934 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:57,934 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:57,934 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:57,934 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. started... 2023-04-19 20:55:57,934 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. as already flushing 2023-04-19 20:55:57,934 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:57,934 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:57,934 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:57,934 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:57,934 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:57,934 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:57,935 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:57,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 296 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35474 deadline: 1681937817928, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:57,936 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:57,936 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:57,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:57,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:57,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:57,936 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:57,936 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:57,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:57,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:57,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:57,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:57,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:57,937 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:57,938 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@182a8011[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:57,938 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:57,938 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:57,939 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:57,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:57,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:57,939 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:57,939 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:57,939 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:57,939 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:57,939 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:57,940 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:57,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:57,940 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:57,940 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:57,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:57,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:57,940 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:57,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:57,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:57,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:57,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:57,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:57,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:57,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:57,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:57,942 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:57,942 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:57,943 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:57,943 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:57,943 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:57,943 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:57,945 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:57,945 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:57,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:57,945 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:57,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:57,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:57,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:57,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:57,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:57,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:57,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:57,946 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:57,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:57,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:57,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:57,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:57,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:57,947 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:57,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:57,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:57,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:57,949 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:57,949 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:57,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:57,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:57,949 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:57,949 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:57,949 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:57,949 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:57,949 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:57,949 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:57,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:57,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:57,949 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:57,949 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:57,949 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:57,949 DEBUG [Thread-3028] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:57,949 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:57,949 DEBUG [Thread-3028] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:57,973 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:57,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 372 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35490 deadline: 1681937817973, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:58,017 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:58,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 267 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35486 deadline: 1681937818013, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:58,018 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:58,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 276 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35470 deadline: 1681937818013, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:58,029 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:58,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 309 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35492 deadline: 1681937818023, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:58,049 DEBUG [Thread-3028] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:58,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:58,052 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:58,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:58,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:58,053 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:58,053 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:58,053 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:58,053 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:58,054 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:58,054 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:58,056 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:58,056 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:58,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:58,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:58,057 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:58,057 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:58,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:58,057 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:58,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:58,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:58,058 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:58,058 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:55:58,058 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:58,058 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:58,058 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:58,058 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:58,058 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:58,058 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:58,059 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:58,059 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. started... 2023-04-19 20:55:58,059 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. as already flushing 2023-04-19 20:55:58,059 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:58,059 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:58,059 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:58,059 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:58,059 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:58,059 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:58,060 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:58,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:58,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:58,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:58,061 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:58,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:58,061 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:58,061 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:58,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:58,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:58,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:58,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:58,062 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:58,062 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@f0c0cbf[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:58,062 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:58,062 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:58,065 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:58,065 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:58,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:58,065 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:58,065 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:58,065 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:58,065 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:58,065 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:58,068 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:58,068 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:58,068 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:58,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:58,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:58,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:58,068 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:58,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:58,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:58,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:58,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:58,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:58,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:58,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:58,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:58,070 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:58,070 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:58,071 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:58,071 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:58,071 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:58,071 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:58,072 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:58,072 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:58,072 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:58,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:58,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:58,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:58,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:58,073 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:58,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:58,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:58,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:58,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:58,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:58,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:58,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:58,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:58,075 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:58,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:58,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:58,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:58,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:58,078 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:58,078 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:58,078 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:58,078 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:58,078 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:58,078 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:58,078 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:58,078 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:58,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:58,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:58,078 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:55:58,078 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:58,078 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:58,078 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:58,078 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:58,078 DEBUG [Thread-3028] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:58,079 DEBUG [Thread-3028] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:58,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:58,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:58,101 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=1057 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/A/302828fd5e1348da88e4f36a38ae9fdb 2023-04-19 20:55:58,179 DEBUG [Thread-3028] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:58,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:58,181 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:58,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:58,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:58,182 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:58,182 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:58,182 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:58,182 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:58,183 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:58,183 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:58,185 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:58,185 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:58,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:58,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:58,185 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:58,185 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:58,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:58,186 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:58,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:58,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:58,186 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:58,186 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:55:58,187 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:58,187 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:58,187 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:58,187 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:58,187 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:58,187 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:58,187 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:58,187 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. started... 2023-04-19 20:55:58,187 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. as already flushing 2023-04-19 20:55:58,187 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:58,187 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:58,187 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:58,187 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:58,187 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:58,187 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:58,189 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:58,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:58,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:58,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:58,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:58,190 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:58,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:58,191 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:58,191 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:58,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:58,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:58,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:58,191 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:58,191 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@a0bb92f[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:58,191 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:58,191 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:58,193 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:58,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:58,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:58,193 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:58,193 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:58,193 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:58,193 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:58,193 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:58,194 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:58,195 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:58,195 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:58,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:58,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:58,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:58,195 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:58,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:58,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:58,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:58,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:58,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:58,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:58,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:58,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:58,199 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:58,199 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:58,199 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:58,199 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:58,199 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:58,199 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:58,201 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:58,201 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:58,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:58,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:58,201 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:58,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:58,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:58,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:58,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:58,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:58,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:58,202 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:58,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:58,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:58,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:58,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:58,206 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:58,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:58,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:58,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:58,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:58,208 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:58,208 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:58,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:58,208 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:58,208 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:55:58,208 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:58,208 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:58,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:58,208 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:58,208 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:58,209 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:58,209 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:58,209 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:58,209 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:58,209 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:58,208 DEBUG [Thread-3028] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:58,209 DEBUG [Thread-3028] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:58,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:58,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:58,309 DEBUG [Thread-3028] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:58,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:58,311 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:58,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:58,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:58,313 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:58,313 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:58,313 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:58,313 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:58,313 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:58,313 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:58,316 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:58,317 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:58,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:58,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:58,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:58,317 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:58,317 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:58,317 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:58,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:58,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:58,318 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:58,318 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:55:58,318 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:58,318 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:58,318 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:58,318 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:58,319 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:58,319 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:58,319 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:58,319 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. started... 2023-04-19 20:55:58,319 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. as already flushing 2023-04-19 20:55:58,319 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:58,319 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:58,319 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:58,319 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:58,319 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:58,319 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:58,322 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:58,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:58,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:58,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:58,322 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:58,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:58,322 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:58,322 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:58,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:58,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:58,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:58,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:58,323 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:58,323 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@52e8bdcd[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:58,324 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:58,324 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:58,325 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:58,325 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:58,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:58,325 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:58,325 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:58,325 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:58,325 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:58,325 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:58,325 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:58,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 278 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35470 deadline: 1681937818319, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:58,326 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:58,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 269 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35486 deadline: 1681937818320, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:58,327 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:58,327 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:58,327 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:58,327 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:58,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:58,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:58,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:58,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:58,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:58,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:58,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:58,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:58,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:58,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:58,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:58,330 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:58,330 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:58,330 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:58,330 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:58,330 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:58,330 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:58,331 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:58,331 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:58,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:58,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:58,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:58,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:58,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:58,331 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:58,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:58,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:58,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:58,332 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:58,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:58,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:58,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:58,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:58,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:58,334 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:58,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:58,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:58,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:58,336 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:58,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 311 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35492 deadline: 1681937818332, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:58,337 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:58,337 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:58,337 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:58,337 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:58,337 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:58,337 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:58,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:58,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:58,337 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:58,337 DEBUG [Thread-3028] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:58,337 DEBUG [Thread-3028] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:58,337 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:58,337 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:58,337 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:58,337 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:58,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:58,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:58,337 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:58,437 DEBUG [Thread-3028] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:58,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:58,440 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:58,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:58,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:58,441 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:58,441 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:58,441 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:58,441 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:58,442 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:58,442 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:58,443 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:58,443 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:58,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:58,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:58,443 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:58,443 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:58,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:58,444 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:58,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:58,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:58,444 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:58,444 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:55:58,444 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:58,445 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:58,445 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:58,445 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:58,445 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:58,445 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:58,445 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:58,445 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. started... 2023-04-19 20:55:58,445 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. as already flushing 2023-04-19 20:55:58,445 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:58,445 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:58,445 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:58,445 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:58,445 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:58,445 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:58,447 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:58,447 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:58,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:58,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:58,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:58,447 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:58,447 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:58,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:58,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:58,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:58,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:58,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:58,449 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:58,449 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e8e8589[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:58,449 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:58,449 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:58,450 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:58,450 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:58,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:58,450 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:58,450 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:58,450 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:58,450 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:58,450 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:58,453 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:58,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:58,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:58,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:58,454 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:58,454 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:58,454 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:58,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:58,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:58,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:58,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:58,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:58,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:58,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:58,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:58,456 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:58,456 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:58,456 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:58,456 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:58,456 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:58,456 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:58,458 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:58,458 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:58,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:58,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:58,458 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:58,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:58,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:58,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:58,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:58,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:58,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:58,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:58,459 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:58,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:58,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:58,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:58,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:58,461 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:58,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:58,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:58,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:58,463 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:58,463 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:58,463 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:58,464 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:58,464 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:58,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:58,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:58,464 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:58,464 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:58,464 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:58,464 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:58,464 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:58,464 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:58,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:58,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:58,464 DEBUG [Thread-3028] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:58,464 DEBUG [Thread-3028] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:58,464 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:58,478 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:58,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 374 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35490 deadline: 1681937818475, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:58,513 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=1057 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/B/533a96acbb4f473fb1601db968733253 2023-04-19 20:55:58,564 DEBUG [Thread-3028] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:58,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:58,567 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:58,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:58,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:58,568 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:58,568 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:58,568 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:58,568 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:58,569 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:58,569 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:58,570 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:58,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:58,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:58,570 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:58,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:58,571 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:58,571 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:58,571 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:58,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:58,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:58,571 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:58,572 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:55:58,572 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:58,572 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:58,572 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:58,572 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:58,572 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:58,572 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:58,572 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:58,572 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. started... 2023-04-19 20:55:58,572 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. as already flushing 2023-04-19 20:55:58,572 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:58,572 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:58,572 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:58,572 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:58,572 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:58,572 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:58,574 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:58,575 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:58,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:58,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:58,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:58,575 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:58,575 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:58,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:58,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:58,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:58,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:58,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:58,576 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:58,576 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@71880b13[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:58,576 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:58,576 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:58,578 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:58,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:58,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:58,578 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:58,578 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:58,578 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:58,578 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:58,578 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:58,579 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:58,579 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:58,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:58,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:58,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:58,579 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:58,580 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:58,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:58,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:58,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:58,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:58,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:58,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:58,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:58,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:58,582 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:58,582 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:58,582 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:58,582 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:58,582 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:58,582 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:58,584 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:58,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:58,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:58,584 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:58,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:58,584 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:58,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:58,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:58,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:58,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:58,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:58,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:58,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:58,585 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:58,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:58,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:58,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:58,586 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:58,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:58,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:58,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:58,590 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:58,590 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:58,590 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:58,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:58,590 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:58,590 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:58,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:58,590 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:58,590 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:58,590 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:58,590 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:58,590 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:58,590 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:58,590 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:58,590 DEBUG [Thread-3028] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:58,591 DEBUG [Thread-3028] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:58,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:58,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:58,691 DEBUG [Thread-3028] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:58,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:58,693 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:58,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:58,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:58,694 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:58,694 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:58,694 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:58,694 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:58,695 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:58,695 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:58,696 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:58,696 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:58,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:58,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:58,696 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:58,696 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:58,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:58,697 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:58,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:58,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:58,697 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:58,700 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:55:58,700 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:58,700 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:58,700 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:58,700 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:58,700 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:58,700 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:58,700 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:58,700 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. started... 2023-04-19 20:55:58,700 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. as already flushing 2023-04-19 20:55:58,700 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:58,701 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:58,701 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:58,701 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:58,701 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:58,701 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:58,703 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:58,703 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:58,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:58,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:58,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:58,704 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:58,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:58,704 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:58,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:58,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:58,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:58,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:58,706 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:58,706 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@261d16cc[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:58,706 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:58,706 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:58,710 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:58,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:58,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:58,711 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:58,711 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:58,711 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:58,711 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:58,711 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:58,712 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:58,712 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:58,712 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:58,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:58,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:58,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:58,712 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:58,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:58,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:58,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:58,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:58,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:58,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:58,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:58,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:58,716 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:58,716 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:58,716 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:58,716 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:58,716 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:58,716 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:58,717 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:58,717 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:58,717 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:58,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:58,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:58,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:58,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:58,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:58,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:58,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:58,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:58,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:58,718 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:58,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:58,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:58,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:58,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:58,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:58,720 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:58,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:58,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:58,722 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:58,722 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:58,722 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:58,722 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:58,722 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:58,722 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:58,722 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:58,722 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:58,722 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:58,722 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:58,722 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:58,722 DEBUG [Thread-3028] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:58,723 DEBUG [Thread-3028] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:58,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:58,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:58,722 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:58,723 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:58,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:58,823 DEBUG [Thread-3028] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:58,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:58,825 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:58,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:58,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:58,826 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:58,826 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:58,826 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:58,826 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:58,826 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:58,826 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:58,828 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:58,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:58,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:58,829 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:58,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:58,829 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:58,829 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:58,829 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:58,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:58,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:58,830 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:58,830 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:55:58,830 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:58,830 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:58,830 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:58,830 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:58,830 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:58,830 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:58,830 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:58,831 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. started... 2023-04-19 20:55:58,831 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. as already flushing 2023-04-19 20:55:58,831 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:58,831 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:58,831 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:58,831 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:58,831 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:58,831 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:58,832 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:58,832 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:58,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:58,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:58,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:58,833 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:58,833 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:58,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:58,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:58,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:58,834 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:58,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 280 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35470 deadline: 1681937818829, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:58,834 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:58,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:58,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 271 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35486 deadline: 1681937818829, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:58,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:58,835 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:58,835 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@318cd117[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:58,835 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:58,835 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:58,836 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:58,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:58,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:58,836 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:58,836 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:58,836 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:58,836 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:58,836 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:58,838 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:58,838 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:58,838 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:58,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:58,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:58,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:58,838 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:58,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:58,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:58,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:58,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:58,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:58,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:58,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:58,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:58,840 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:58,840 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:58,840 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:58,840 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:58,840 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:58,840 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:58,842 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:58,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 313 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35492 deadline: 1681937818838, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:58,843 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:58,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:58,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:58,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:58,844 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:58,844 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:58,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:58,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:58,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:58,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:58,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:58,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:58,844 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:58,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:58,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:58,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:58,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:58,846 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:58,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:58,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:58,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:58,848 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:58,848 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:58,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:58,848 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:58,849 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:58,848 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:58,848 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:58,849 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:58,849 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:58,849 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:58,849 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:58,849 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:58,848 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:58,849 DEBUG [Thread-3028] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:58,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:58,849 DEBUG [Thread-3028] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:58,849 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:58,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:58,929 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=1057 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/C/2dc58269c48d45a0a10d64830673f6d1 2023-04-19 20:55:58,932 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/A/302828fd5e1348da88e4f36a38ae9fdb as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/302828fd5e1348da88e4f36a38ae9fdb 2023-04-19 20:55:58,936 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/302828fd5e1348da88e4f36a38ae9fdb, entries=200, sequenceid=1057, filesize=14.3 K 2023-04-19 20:55:58,937 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/B/533a96acbb4f473fb1601db968733253 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/533a96acbb4f473fb1601db968733253 2023-04-19 20:55:58,940 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/533a96acbb4f473fb1601db968733253, entries=150, sequenceid=1057, filesize=11.9 K 2023-04-19 20:55:58,941 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/C/2dc58269c48d45a0a10d64830673f6d1 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/2dc58269c48d45a0a10d64830673f6d1 2023-04-19 20:55:58,943 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/2dc58269c48d45a0a10d64830673f6d1, entries=150, sequenceid=1057, filesize=11.9 K 2023-04-19 20:55:58,944 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~390.80 KB/400176, currentSize=46.96 KB/48090 for 4393734ae917e967cf2b5b4b3b44ed0d in 1266ms, sequenceid=1057, compaction requested=true 2023-04-19 20:55:58,944 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 4393734ae917e967cf2b5b4b3b44ed0d: 2023-04-19 20:55:58,944 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 20:55:58,944 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-19 20:55:58,944 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-19 20:55:58,944 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 20:55:58,944 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 20:55:58,946 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 50242 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-19 20:55:58,946 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1912): 4393734ae917e967cf2b5b4b3b44ed0d/B is initiating minor compaction (all files) 2023-04-19 20:55:58,946 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 52682 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-19 20:55:58,946 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4393734ae917e967cf2b5b4b3b44ed0d/B in TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:58,946 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1912): 4393734ae917e967cf2b5b4b3b44ed0d/A is initiating minor compaction (all files) 2023-04-19 20:55:58,946 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/f300d42c7c5048619332277a7d304dae, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/b2cf2936dbe54253b08855f45730c8be, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/7b941f46c179455e9ed81fe241ff0761, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/533a96acbb4f473fb1601db968733253] into tmpdir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp, totalSize=49.1 K 2023-04-19 20:55:58,946 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 4393734ae917e967cf2b5b4b3b44ed0d/A in TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:58,946 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/548fac84597544b7aa3353b95607a054, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/fd79b6156cd94489820ffcc7eada3452, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/318be65f80634fa0b4defcc935446002, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/302828fd5e1348da88e4f36a38ae9fdb] into tmpdir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp, totalSize=51.4 K 2023-04-19 20:55:58,947 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting f300d42c7c5048619332277a7d304dae, keycount=150, bloomtype=ROW, size=13.4 K, encoding=NONE, compression=NONE, seqNum=955, earliestPutTs=1681937755790 2023-04-19 20:55:58,947 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 548fac84597544b7aa3353b95607a054, keycount=150, bloomtype=ROW, size=13.4 K, encoding=NONE, compression=NONE, seqNum=955, earliestPutTs=1681937755790 2023-04-19 20:55:58,947 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting b2cf2936dbe54253b08855f45730c8be, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1009, earliestPutTs=1681937756323 2023-04-19 20:55:58,947 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting fd79b6156cd94489820ffcc7eada3452, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1009, earliestPutTs=1681937756323 2023-04-19 20:55:58,947 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 7b941f46c179455e9ed81fe241ff0761, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1022, earliestPutTs=1681937756666 2023-04-19 20:55:58,948 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 318be65f80634fa0b4defcc935446002, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1022, earliestPutTs=1681937756666 2023-04-19 20:55:58,948 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 533a96acbb4f473fb1601db968733253, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1057, earliestPutTs=1681937757353 2023-04-19 20:55:58,948 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 302828fd5e1348da88e4f36a38ae9fdb, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=1057, earliestPutTs=1681937757332 2023-04-19 20:55:58,949 DEBUG [Thread-3028] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:58,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:58,952 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:58,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:58,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:58,953 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:58,954 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:58,954 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:58,954 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:58,954 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:58,954 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:58,955 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:58,955 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:58,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:58,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:58,956 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:58,956 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:55:58,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:58,956 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:58,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:55:58,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:58,956 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:55:58,957 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:55:58,957 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:55:58,957 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:55:58,957 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:58,957 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:55:58,959 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] throttle.PressureAwareThroughputController(145): 4393734ae917e967cf2b5b4b3b44ed0d#B#compaction#525 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 20:55:58,964 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:55:58,964 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:55:58,966 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] throttle.PressureAwareThroughputController(145): 4393734ae917e967cf2b5b4b3b44ed0d#A#compaction#526 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 20:55:58,968 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:58,968 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. started... 2023-04-19 20:55:58,968 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4393734ae917e967cf2b5b4b3b44ed0d 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-19 20:55:58,968 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4393734ae917e967cf2b5b4b3b44ed0d, store=A 2023-04-19 20:55:58,968 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:55:58,969 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4393734ae917e967cf2b5b4b3b44ed0d, store=B 2023-04-19 20:55:58,969 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:55:58,969 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4393734ae917e967cf2b5b4b3b44ed0d, store=C 2023-04-19 20:55:58,969 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:55:59,389 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/B/717b44d64f824940b5bb5a04ca662a12 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/717b44d64f824940b5bb5a04ca662a12 2023-04-19 20:55:59,390 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/A/f9b00f77845a4984ba7f60304d5ec246 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/f9b00f77845a4984ba7f60304d5ec246 2023-04-19 20:55:59,393 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 4393734ae917e967cf2b5b4b3b44ed0d/B of 4393734ae917e967cf2b5b4b3b44ed0d into 717b44d64f824940b5bb5a04ca662a12(size=13.5 K), total size for store is 13.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 20:55:59,393 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4393734ae917e967cf2b5b4b3b44ed0d: 2023-04-19 20:55:59,394 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d., storeName=4393734ae917e967cf2b5b4b3b44ed0d/B, priority=12, startTime=1681937758944; duration=0sec 2023-04-19 20:55:59,394 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 20:55:59,394 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-19 20:55:59,394 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=1067 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/A/db41dc240c404d13b7e942b99c06d4a7 2023-04-19 20:55:59,395 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 50174 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-19 20:55:59,395 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1912): 4393734ae917e967cf2b5b4b3b44ed0d/C is initiating minor compaction (all files) 2023-04-19 20:55:59,396 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4393734ae917e967cf2b5b4b3b44ed0d/C in TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:59,396 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/b634153cab4148388e9e607cbbdcdf58, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/5397c2fbad0249d4a559380e55097d25, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/ff31eae303cc4f7f83a58ec6b50c69ea, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/2dc58269c48d45a0a10d64830673f6d1] into tmpdir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp, totalSize=49.0 K 2023-04-19 20:55:59,396 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting b634153cab4148388e9e607cbbdcdf58, keycount=150, bloomtype=ROW, size=13.3 K, encoding=NONE, compression=NONE, seqNum=955, earliestPutTs=1681937755790 2023-04-19 20:55:59,396 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 4393734ae917e967cf2b5b4b3b44ed0d/A of 4393734ae917e967cf2b5b4b3b44ed0d into f9b00f77845a4984ba7f60304d5ec246(size=13.5 K), total size for store is 13.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 20:55:59,396 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4393734ae917e967cf2b5b4b3b44ed0d: 2023-04-19 20:55:59,397 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d., storeName=4393734ae917e967cf2b5b4b3b44ed0d/A, priority=12, startTime=1681937758944; duration=0sec 2023-04-19 20:55:59,397 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 5397c2fbad0249d4a559380e55097d25, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1009, earliestPutTs=1681937756323 2023-04-19 20:55:59,397 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 20:55:59,397 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting ff31eae303cc4f7f83a58ec6b50c69ea, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1022, earliestPutTs=1681937756666 2023-04-19 20:55:59,398 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 2dc58269c48d45a0a10d64830673f6d1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1057, earliestPutTs=1681937757353 2023-04-19 20:55:59,413 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] throttle.PressureAwareThroughputController(145): 4393734ae917e967cf2b5b4b3b44ed0d#C#compaction#529 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 20:55:59,432 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=1067 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/B/ac03180f8ee24999a82b039fab20977c 2023-04-19 20:55:59,448 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/C/4462551c90984908a8869f965accf87b as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/4462551c90984908a8869f965accf87b 2023-04-19 20:55:59,455 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 4393734ae917e967cf2b5b4b3b44ed0d/C of 4393734ae917e967cf2b5b4b3b44ed0d into 4462551c90984908a8869f965accf87b(size=13.4 K), total size for store is 13.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 20:55:59,455 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4393734ae917e967cf2b5b4b3b44ed0d: 2023-04-19 20:55:59,455 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d., storeName=4393734ae917e967cf2b5b4b3b44ed0d/C, priority=12, startTime=1681937758944; duration=0sec 2023-04-19 20:55:59,456 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 20:55:59,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(9158): Flush requested on 4393734ae917e967cf2b5b4b3b44ed0d 2023-04-19 20:55:59,487 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. as already flushing 2023-04-19 20:55:59,579 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681937759543, max timestamp=1681937759577Num uniques 859; 2023-04-19 20:55:59,579 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681937759543, max timestamp=1681937759577Num uniques 859; 2023-04-19 20:55:59,579 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 20:55:59,579 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 20:55:59,579 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681937759543, max timestamp=1681937759577Num uniques 859; 2023-04-19 20:55:59,579 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 20:55:59,664 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:59,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 410 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35490 deadline: 1681937819654, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:59,769 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:59,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 412 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35490 deadline: 1681937819765, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:59,848 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:59,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 282 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35470 deadline: 1681937819843, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:59,851 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:59,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 273 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35486 deadline: 1681937819844, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:59,853 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:59,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 315 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35492 deadline: 1681937819850, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:59,862 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=1067 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/C/e9451d05b033439bba2b444d75b7dd54 2023-04-19 20:55:59,866 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/A/db41dc240c404d13b7e942b99c06d4a7 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/db41dc240c404d13b7e942b99c06d4a7 2023-04-19 20:55:59,870 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/db41dc240c404d13b7e942b99c06d4a7, entries=150, sequenceid=1067, filesize=11.9 K 2023-04-19 20:55:59,870 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/B/ac03180f8ee24999a82b039fab20977c as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/ac03180f8ee24999a82b039fab20977c 2023-04-19 20:55:59,874 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/ac03180f8ee24999a82b039fab20977c, entries=150, sequenceid=1067, filesize=11.9 K 2023-04-19 20:55:59,874 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/C/e9451d05b033439bba2b444d75b7dd54 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/e9451d05b033439bba2b444d75b7dd54 2023-04-19 20:55:59,878 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/e9451d05b033439bba2b444d75b7dd54, entries=150, sequenceid=1067, filesize=11.9 K 2023-04-19 20:55:59,878 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 4393734ae917e967cf2b5b4b3b44ed0d in 910ms, sequenceid=1067, compaction requested=false 2023-04-19 20:55:59,878 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4393734ae917e967cf2b5b4b3b44ed0d: 2023-04-19 20:55:59,878 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:55:59,878 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:55:59,878 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:55:59,878 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:59,879 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:55:59,879 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:55:59,880 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:59,880 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:59,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:59,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:59,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:59,880 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:59,880 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:55:59,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:59,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:59,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:59,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:59,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:59,882 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:55:59,882 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5bc4b862[Count = 0] remaining members to acquire global barrier 2023-04-19 20:55:59,882 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:55:59,882 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:59,883 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:59,883 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:59,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:59,883 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:55:59,883 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:55:59,883 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:55:59,883 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:59,883 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:55:59,884 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:55:59,885 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:59,885 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:59,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:59,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:59,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:59,885 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:55:59,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:59,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:59,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:59,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:59,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:59,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:59,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:59,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:55:59,887 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:55:59,887 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:55:59,887 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:55:59,887 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:55:59,887 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:55:59,887 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:55:59,890 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:59,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:59,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:55:59,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:55:59,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:55:59,890 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:59,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:59,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:55:59,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:59,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:59,892 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:59,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:55:59,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:55:59,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:59,894 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:59,894 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:55:59,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:59,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:59,894 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:59,894 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:59,894 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:55:59,894 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:59,895 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:59,895 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:55:59,895 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:55:59,895 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:55:59,895 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:55:59,895 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:55:59,895 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:55:59,895 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:59,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:55:59,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:55:59,895 DEBUG [Thread-3028] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:55:59,895 DEBUG [Thread-3028] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:55:59,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:55:59,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:55:59,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(9158): Flush requested on 4393734ae917e967cf2b5b4b3b44ed0d 2023-04-19 20:55:59,955 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 4393734ae917e967cf2b5b4b3b44ed0d 3/3 column families, dataSize=140.89 KB heapSize=405.61 KB 2023-04-19 20:55:59,955 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4393734ae917e967cf2b5b4b3b44ed0d, store=A 2023-04-19 20:55:59,955 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-19 20:55:59,955 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4393734ae917e967cf2b5b4b3b44ed0d, store=B 2023-04-19 20:55:59,955 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-19 20:55:59,955 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4393734ae917e967cf2b5b4b3b44ed0d, store=C 2023-04-19 20:55:59,955 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-19 20:55:59,985 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:59,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 303 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35474 deadline: 1681937819980, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:59,991 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:55:59,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 416 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35490 deadline: 1681937819984, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:55:59,995 DEBUG [Thread-3028] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:55:59,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:55:59,997 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:55:59,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:55:59,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:55:59,998 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:55:59,998 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:55:59,998 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:55:59,998 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:55:59,998 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:55:59,998 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:00,000 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:56:00,000 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:00,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:56:00,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:56:00,000 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:00,000 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:56:00,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:00,001 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:00,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:56:00,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:00,001 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:56:00,001 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:56:00,001 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:56:00,001 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:56:00,001 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:56:00,001 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:56:00,002 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:56:00,002 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:56:00,002 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:56:00,002 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. started... 2023-04-19 20:56:00,002 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. as already flushing 2023-04-19 20:56:00,002 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:56:00,002 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:56:00,002 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:56:00,002 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:00,002 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:56:00,002 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:56:00,003 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:00,003 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:00,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:00,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:00,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:00,004 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:00,004 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:56:00,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:00,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:00,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:00,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:00,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:00,005 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:56:00,005 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6aa5e09a[Count = 0] remaining members to acquire global barrier 2023-04-19 20:56:00,005 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:56:00,005 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:00,007 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:00,007 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:00,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:00,007 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:56:00,007 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:56:00,007 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:56:00,007 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:00,008 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:56:00,012 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:00,012 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:56:00,012 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:56:00,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:00,012 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:56:00,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:00,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:00,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:00,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:00,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:00,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:00,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:00,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:00,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:00,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:56:00,014 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:56:00,014 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:56:00,014 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:56:00,014 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:56:00,014 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:56:00,014 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:56:00,016 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:00,016 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:00,016 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:56:00,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:00,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:00,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:00,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:00,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:00,016 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:00,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:56:00,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:56:00,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:00,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:00,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:00,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:00,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:00,017 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:00,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:00,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:00,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:00,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:00,021 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:00,021 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:56:00,021 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:00,021 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:00,021 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:56:00,021 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:56:00,021 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:56:00,021 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:56:00,021 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:00,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:56:00,021 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:00,022 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:00,022 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:00,021 DEBUG [Thread-3028] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:56:00,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:56:00,022 DEBUG [Thread-3028] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:56:00,022 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:56:00,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:56:00,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:56:00,090 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:00,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 305 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35474 deadline: 1681937820086, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:00,100 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:00,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 418 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35490 deadline: 1681937820092, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:00,122 DEBUG [Thread-3028] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:56:00,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:56:00,124 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:56:00,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:56:00,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:56:00,128 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:56:00,128 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:56:00,128 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:56:00,128 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:56:00,129 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:00,129 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:00,130 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:00,130 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:56:00,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:56:00,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:56:00,130 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:00,130 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:56:00,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:00,130 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:00,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:56:00,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:00,131 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:56:00,131 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:56:00,131 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:56:00,131 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:56:00,131 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:56:00,131 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:56:00,131 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:56:00,131 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:56:00,132 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:56:00,132 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. started... 2023-04-19 20:56:00,132 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. as already flushing 2023-04-19 20:56:00,132 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:56:00,132 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:56:00,132 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:56:00,132 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:00,132 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:56:00,132 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:56:00,134 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:00,134 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:00,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:00,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:00,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:00,134 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:00,134 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:56:00,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:00,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:00,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:00,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:00,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:00,136 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:56:00,136 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@24403074[Count = 0] remaining members to acquire global barrier 2023-04-19 20:56:00,136 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:56:00,136 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:00,137 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:00,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:00,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:00,137 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:56:00,137 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:56:00,137 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:56:00,137 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:00,137 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:56:00,141 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:00,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:00,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:00,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:00,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:00,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:00,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:00,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:00,142 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:56:00,142 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:56:00,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:00,142 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:56:00,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:00,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:00,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:56:00,145 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:56:00,145 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:56:00,145 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:56:00,145 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:56:00,145 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:56:00,145 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:56:00,147 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:00,147 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:00,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:00,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:00,147 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:56:00,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:00,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:00,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:00,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:56:00,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:56:00,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:00,147 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:00,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:00,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:00,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:00,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:00,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:00,150 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:00,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:00,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:00,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:00,152 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:00,153 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:00,153 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:56:00,153 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:00,153 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:56:00,153 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:00,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:56:00,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:56:00,153 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:00,153 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:56:00,153 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:00,153 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:00,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:56:00,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:56:00,154 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:56:00,154 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:56:00,154 DEBUG [Thread-3028] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:56:00,154 DEBUG [Thread-3028] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:56:00,254 DEBUG [Thread-3028] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:56:00,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:56:00,255 DEBUG [Thread-3033] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x15f74c78 to 127.0.0.1:58736 2023-04-19 20:56:00,256 DEBUG [Thread-3033] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-19 20:56:00,260 DEBUG [Thread-3029] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7a4e3b2e to 127.0.0.1:58736 2023-04-19 20:56:00,260 DEBUG [Thread-3029] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-19 20:56:00,261 DEBUG [Thread-3031] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1f65627a to 127.0.0.1:58736 2023-04-19 20:56:00,261 DEBUG [Thread-3031] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-19 20:56:00,267 DEBUG [Thread-3037] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1f6e865f to 127.0.0.1:58736 2023-04-19 20:56:00,267 DEBUG [Thread-3037] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-19 20:56:00,270 DEBUG [Thread-3035] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1401cac1 to 127.0.0.1:58736 2023-04-19 20:56:00,270 DEBUG [Thread-3035] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-19 20:56:00,293 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:00,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 307 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35474 deadline: 1681937820292, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:00,302 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:00,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 420 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35490 deadline: 1681937820302, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:00,364 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=1108 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/A/10f4691964204c6888d0f9d66c5d781c 2023-04-19 20:56:00,596 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:00,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 309 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35474 deadline: 1681937820596, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:00,604 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:00,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 422 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35490 deadline: 1681937820604, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:00,772 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=1108 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/B/473c92cb23c342eb84038b54d357ef30 2023-04-19 20:56:01,101 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:01,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 311 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35474 deadline: 1681937821101, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:01,109 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:01,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 424 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35490 deadline: 1681937821109, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:01,180 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=1108 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/C/d5086dcd74f0450497e1768e1b0c6afd 2023-04-19 20:56:01,184 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/A/10f4691964204c6888d0f9d66c5d781c as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/10f4691964204c6888d0f9d66c5d781c 2023-04-19 20:56:01,186 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/10f4691964204c6888d0f9d66c5d781c, entries=200, sequenceid=1108, filesize=14.3 K 2023-04-19 20:56:01,187 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/B/473c92cb23c342eb84038b54d357ef30 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/473c92cb23c342eb84038b54d357ef30 2023-04-19 20:56:01,190 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/473c92cb23c342eb84038b54d357ef30, entries=150, sequenceid=1108, filesize=11.9 K 2023-04-19 20:56:01,190 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/C/d5086dcd74f0450497e1768e1b0c6afd as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/d5086dcd74f0450497e1768e1b0c6afd 2023-04-19 20:56:01,193 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/d5086dcd74f0450497e1768e1b0c6afd, entries=150, sequenceid=1108, filesize=11.9 K 2023-04-19 20:56:01,193 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~405.56 KB/415296, currentSize=40.25 KB/41220 for 4393734ae917e967cf2b5b4b3b44ed0d in 1238ms, sequenceid=1108, compaction requested=true 2023-04-19 20:56:01,193 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 4393734ae917e967cf2b5b4b3b44ed0d: 2023-04-19 20:56:01,194 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 20:56:01,194 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-19 20:56:01,194 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 20:56:01,194 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-19 20:56:01,194 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 20:56:01,194 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 40632 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-19 20:56:01,194 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1912): 4393734ae917e967cf2b5b4b3b44ed0d/A is initiating minor compaction (all files) 2023-04-19 20:56:01,195 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 38192 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-19 20:56:01,195 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 4393734ae917e967cf2b5b4b3b44ed0d/A in TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:56:01,195 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1912): 4393734ae917e967cf2b5b4b3b44ed0d/B is initiating minor compaction (all files) 2023-04-19 20:56:01,195 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/f9b00f77845a4984ba7f60304d5ec246, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/db41dc240c404d13b7e942b99c06d4a7, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/10f4691964204c6888d0f9d66c5d781c] into tmpdir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp, totalSize=39.7 K 2023-04-19 20:56:01,195 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4393734ae917e967cf2b5b4b3b44ed0d/B in TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:56:01,195 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/717b44d64f824940b5bb5a04ca662a12, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/ac03180f8ee24999a82b039fab20977c, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/473c92cb23c342eb84038b54d357ef30] into tmpdir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp, totalSize=37.3 K 2023-04-19 20:56:01,195 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting f9b00f77845a4984ba7f60304d5ec246, keycount=150, bloomtype=ROW, size=13.5 K, encoding=NONE, compression=NONE, seqNum=1057, earliestPutTs=1681937757353 2023-04-19 20:56:01,195 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 717b44d64f824940b5bb5a04ca662a12, keycount=150, bloomtype=ROW, size=13.5 K, encoding=NONE, compression=NONE, seqNum=1057, earliestPutTs=1681937757353 2023-04-19 20:56:01,195 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting db41dc240c404d13b7e942b99c06d4a7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1067, earliestPutTs=1681937757686 2023-04-19 20:56:01,195 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting ac03180f8ee24999a82b039fab20977c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1067, earliestPutTs=1681937757686 2023-04-19 20:56:01,195 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 10f4691964204c6888d0f9d66c5d781c, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=1108, earliestPutTs=1681937759591 2023-04-19 20:56:01,195 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 473c92cb23c342eb84038b54d357ef30, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1108, earliestPutTs=1681937759640 2023-04-19 20:56:01,200 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] throttle.PressureAwareThroughputController(145): 4393734ae917e967cf2b5b4b3b44ed0d#A#compaction#534 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 20:56:01,200 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] throttle.PressureAwareThroughputController(145): 4393734ae917e967cf2b5b4b3b44ed0d#B#compaction#535 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 20:56:01,607 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/B/fb50241348e240d18b98afb69ce56b21 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/fb50241348e240d18b98afb69ce56b21 2023-04-19 20:56:01,608 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/A/9f18b0eb5747402fa6ee1b1a6064262b as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/9f18b0eb5747402fa6ee1b1a6064262b 2023-04-19 20:56:01,611 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 4393734ae917e967cf2b5b4b3b44ed0d/B of 4393734ae917e967cf2b5b4b3b44ed0d into fb50241348e240d18b98afb69ce56b21(size=13.6 K), total size for store is 13.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 20:56:01,611 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4393734ae917e967cf2b5b4b3b44ed0d: 2023-04-19 20:56:01,611 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d., storeName=4393734ae917e967cf2b5b4b3b44ed0d/B, priority=13, startTime=1681937761194; duration=0sec 2023-04-19 20:56:01,611 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 4393734ae917e967cf2b5b4b3b44ed0d/A of 4393734ae917e967cf2b5b4b3b44ed0d into 9f18b0eb5747402fa6ee1b1a6064262b(size=13.6 K), total size for store is 13.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 20:56:01,611 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 20:56:01,611 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4393734ae917e967cf2b5b4b3b44ed0d: 2023-04-19 20:56:01,611 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-19 20:56:01,611 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d., storeName=4393734ae917e967cf2b5b4b3b44ed0d/A, priority=13, startTime=1681937761194; duration=0sec 2023-04-19 20:56:01,611 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 20:56:01,612 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 38124 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-19 20:56:01,612 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1912): 4393734ae917e967cf2b5b4b3b44ed0d/C is initiating minor compaction (all files) 2023-04-19 20:56:01,612 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4393734ae917e967cf2b5b4b3b44ed0d/C in TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:56:01,612 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/4462551c90984908a8869f965accf87b, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/e9451d05b033439bba2b444d75b7dd54, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/d5086dcd74f0450497e1768e1b0c6afd] into tmpdir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp, totalSize=37.2 K 2023-04-19 20:56:01,612 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 4462551c90984908a8869f965accf87b, keycount=150, bloomtype=ROW, size=13.4 K, encoding=NONE, compression=NONE, seqNum=1057, earliestPutTs=1681937757353 2023-04-19 20:56:01,612 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting e9451d05b033439bba2b444d75b7dd54, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1067, earliestPutTs=1681937757686 2023-04-19 20:56:01,613 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting d5086dcd74f0450497e1768e1b0c6afd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1108, earliestPutTs=1681937759640 2023-04-19 20:56:01,617 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] throttle.PressureAwareThroughputController(145): 4393734ae917e967cf2b5b4b3b44ed0d#C#compaction#536 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 20:56:01,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(9158): Flush requested on 4393734ae917e967cf2b5b4b3b44ed0d 2023-04-19 20:56:01,856 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 4393734ae917e967cf2b5b4b3b44ed0d 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-19 20:56:01,856 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4393734ae917e967cf2b5b4b3b44ed0d, store=A 2023-04-19 20:56:01,856 DEBUG [Thread-3026] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x50150525 to 127.0.0.1:58736 2023-04-19 20:56:01,856 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:56:01,856 DEBUG [Thread-3026] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-19 20:56:01,856 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4393734ae917e967cf2b5b4b3b44ed0d, store=B 2023-04-19 20:56:01,856 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:56:01,856 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4393734ae917e967cf2b5b4b3b44ed0d, store=C 2023-04-19 20:56:01,856 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:56:01,865 DEBUG [Thread-3024] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3816bdf5 to 127.0.0.1:58736 2023-04-19 20:56:01,865 DEBUG [Thread-3024] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-19 20:56:01,866 DEBUG [Thread-3022] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x15840296 to 127.0.0.1:58736 2023-04-19 20:56:01,866 DEBUG [Thread-3022] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-19 20:56:02,024 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/C/23d714a08c8f41518adfdf2581ad1285 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/23d714a08c8f41518adfdf2581ad1285 2023-04-19 20:56:02,027 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 4393734ae917e967cf2b5b4b3b44ed0d/C of 4393734ae917e967cf2b5b4b3b44ed0d into 23d714a08c8f41518adfdf2581ad1285(size=13.5 K), total size for store is 13.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 20:56:02,028 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4393734ae917e967cf2b5b4b3b44ed0d: 2023-04-19 20:56:02,028 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d., storeName=4393734ae917e967cf2b5b4b3b44ed0d/C, priority=13, startTime=1681937761194; duration=0sec 2023-04-19 20:56:02,028 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 20:56:02,111 DEBUG [Thread-3020] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2ab56e54 to 127.0.0.1:58736 2023-04-19 20:56:02,111 DEBUG [Thread-3020] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-19 20:56:02,116 DEBUG [Thread-3018] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6e0b6187 to 127.0.0.1:58736 2023-04-19 20:56:02,116 DEBUG [Thread-3018] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-19 20:56:02,117 INFO [Listener at localhost/34893] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-19 20:56:02,117 INFO [Listener at localhost/34893] hbase.AcidGuaranteesTestTool(394): wrote 279 2023-04-19 20:56:02,117 INFO [Listener at localhost/34893] hbase.AcidGuaranteesTestTool(394): wrote 184 2023-04-19 20:56:02,117 INFO [Listener at localhost/34893] hbase.AcidGuaranteesTestTool(394): wrote 153 2023-04-19 20:56:02,117 INFO [Listener at localhost/34893] hbase.AcidGuaranteesTestTool(394): wrote 180 2023-04-19 20:56:02,117 INFO [Listener at localhost/34893] hbase.AcidGuaranteesTestTool(394): wrote 170 2023-04-19 20:56:02,117 INFO [Listener at localhost/34893] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-19 20:56:02,117 INFO [Listener at localhost/34893] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-19 20:56:02,117 INFO [Listener at localhost/34893] hbase.AcidGuaranteesTestTool(402): scanned 1649 2023-04-19 20:56:02,117 INFO [Listener at localhost/34893] hbase.AcidGuaranteesTestTool(403): verified 4947 rows 2023-04-19 20:56:02,117 INFO [Listener at localhost/34893] hbase.AcidGuaranteesTestTool(402): scanned 1651 2023-04-19 20:56:02,117 INFO [Listener at localhost/34893] hbase.AcidGuaranteesTestTool(403): verified 4951 rows 2023-04-19 20:56:02,117 INFO [Listener at localhost/34893] hbase.AcidGuaranteesTestTool(402): scanned 1650 2023-04-19 20:56:02,117 INFO [Listener at localhost/34893] hbase.AcidGuaranteesTestTool(403): verified 4950 rows 2023-04-19 20:56:02,117 INFO [Listener at localhost/34893] hbase.AcidGuaranteesTestTool(402): scanned 1647 2023-04-19 20:56:02,117 INFO [Listener at localhost/34893] hbase.AcidGuaranteesTestTool(403): verified 4941 rows 2023-04-19 20:56:02,117 INFO [Listener at localhost/34893] hbase.AcidGuaranteesTestTool(402): scanned 1653 2023-04-19 20:56:02,117 INFO [Listener at localhost/34893] hbase.AcidGuaranteesTestTool(403): verified 4959 rows 2023-04-19 20:56:02,117 INFO [Listener at localhost/34893] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-19 20:56:02,118 DEBUG [Listener at localhost/34893] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x66369677 to 127.0.0.1:58736 2023-04-19 20:56:02,118 DEBUG [Listener at localhost/34893] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-19 20:56:02,119 INFO [Listener at localhost/34893] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-19 20:56:02,119 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-19 20:56:02,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure2.ProcedureExecutor(1029): Stored pid=50, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-19 20:56:02,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-19 20:56:02,123 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681937762123"}]},"ts":"1681937762123"} 2023-04-19 20:56:02,124 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-19 20:56:02,126 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-19 20:56:02,127 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=51, ppid=50, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=4393734ae917e967cf2b5b4b3b44ed0d, UNASSIGN}] 2023-04-19 20:56:02,128 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=51, ppid=50, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=4393734ae917e967cf2b5b4b3b44ed0d, UNASSIGN 2023-04-19 20:56:02,129 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=4393734ae917e967cf2b5b4b3b44ed0d, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:02,129 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681937762129"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681937762129"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681937762129"}]},"ts":"1681937762129"} 2023-04-19 20:56:02,129 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=52, ppid=51, state=RUNNABLE; CloseRegionProcedure 4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971}] 2023-04-19 20:56:02,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-19 20:56:02,262 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=1120 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/A/b695deeeb77b4b9caadb1846dc79dd44 2023-04-19 20:56:02,270 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=1120 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/B/bf4a2cc56ba9479fa8cbdbc30b143f78 2023-04-19 20:56:02,281 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 4393734ae917e967cf2b5b4b3b44ed0d 2023-04-19 20:56:02,281 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 4393734ae917e967cf2b5b4b3b44ed0d, disabling compactions & flushes 2023-04-19 20:56:02,281 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1858): waiting for 0 compactions & cache flush to complete for region TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:56:02,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-19 20:56:02,677 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=1120 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/C/332aaadde0174cc2a72993ce24de6c9a 2023-04-19 20:56:02,681 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/A/b695deeeb77b4b9caadb1846dc79dd44 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/b695deeeb77b4b9caadb1846dc79dd44 2023-04-19 20:56:02,683 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/b695deeeb77b4b9caadb1846dc79dd44, entries=150, sequenceid=1120, filesize=11.9 K 2023-04-19 20:56:02,684 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/B/bf4a2cc56ba9479fa8cbdbc30b143f78 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/bf4a2cc56ba9479fa8cbdbc30b143f78 2023-04-19 20:56:02,687 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/bf4a2cc56ba9479fa8cbdbc30b143f78, entries=150, sequenceid=1120, filesize=11.9 K 2023-04-19 20:56:02,687 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/C/332aaadde0174cc2a72993ce24de6c9a as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/332aaadde0174cc2a72993ce24de6c9a 2023-04-19 20:56:02,691 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/332aaadde0174cc2a72993ce24de6c9a, entries=150, sequenceid=1120, filesize=11.9 K 2023-04-19 20:56:02,691 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=26.84 KB/27480 for 4393734ae917e967cf2b5b4b3b44ed0d in 835ms, sequenceid=1120, compaction requested=false 2023-04-19 20:56:02,691 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 4393734ae917e967cf2b5b4b3b44ed0d: 2023-04-19 20:56:02,691 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:56:02,691 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:56:02,691 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. after waiting 0 ms 2023-04-19 20:56:02,691 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:56:02,692 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 4393734ae917e967cf2b5b4b3b44ed0d 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-19 20:56:02,692 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4393734ae917e967cf2b5b4b3b44ed0d, store=A 2023-04-19 20:56:02,692 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:56:02,692 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4393734ae917e967cf2b5b4b3b44ed0d, store=B 2023-04-19 20:56:02,692 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:56:02,692 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4393734ae917e967cf2b5b4b3b44ed0d, store=C 2023-04-19 20:56:02,692 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:56:02,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-19 20:56:03,098 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=1128 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/A/a6e8d7d34347445cbc42b90de7a98cb7 2023-04-19 20:56:03,106 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=1128 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/B/65e068f4295547ad99d13aee864dd010 2023-04-19 20:56:03,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-19 20:56:03,514 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=1128 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/C/52688eb791c445f2b2868fbcf00586c9 2023-04-19 20:56:03,517 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/A/a6e8d7d34347445cbc42b90de7a98cb7 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/a6e8d7d34347445cbc42b90de7a98cb7 2023-04-19 20:56:03,520 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/a6e8d7d34347445cbc42b90de7a98cb7, entries=100, sequenceid=1128, filesize=9.5 K 2023-04-19 20:56:03,521 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/B/65e068f4295547ad99d13aee864dd010 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/65e068f4295547ad99d13aee864dd010 2023-04-19 20:56:03,524 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/65e068f4295547ad99d13aee864dd010, entries=100, sequenceid=1128, filesize=9.5 K 2023-04-19 20:56:03,524 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/.tmp/C/52688eb791c445f2b2868fbcf00586c9 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/52688eb791c445f2b2868fbcf00586c9 2023-04-19 20:56:03,527 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/52688eb791c445f2b2868fbcf00586c9, entries=100, sequenceid=1128, filesize=9.5 K 2023-04-19 20:56:03,528 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for 4393734ae917e967cf2b5b4b3b44ed0d in 836ms, sequenceid=1128, compaction requested=true 2023-04-19 20:56:03,551 DEBUG [StoreCloser-TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/593cad6360284a51a3288515b09116e0, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/0e6df69316b3476995b3bbd854a55c3f, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/6508ddb2cff04e129cbb9f90ca8b10ef, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/fc4621db25f14d989e8f3460cfd0d42f, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/f19b87eaae6941969bb821a857ca9dc8, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/115eea192f7b4b21a872c8f0c75a8813, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/b60ee764ac8e437091ed5a69bfaa3b13, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/e87262c540ef4372b3ec73f498a60b28, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/bdea0565ed934118b46236767cc2487b, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/9ea01a31f7cb415ebbef0aa052832786, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/9a91bef8a8544db6a38a37d8b1eb6d58, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/3ffc60b4c21d4ccd8c01900796cac8cf, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/0f1c5c4ed0b14221b9e65af5aaa0e473, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/15edd9481e0c4eb4a4e6176a7acb7ae8, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/66568d2c0e694ec59e2758f088f94f96, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/785eec2bf2554d30b5c16f9a25379568, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/46d649f1d4c64cbbaad84dd81d645bac, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/83f1252ec5f44d1585561d734723ea22, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/476f2dffd0a0400d8b8e9e7891445841, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/3d713cd6cad44bf8909494a274ee3256, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/1b0d4fd1deee4abab035f01ee9d1d0c0, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/f3aed609eeee412dbad9a1322a48bb8f, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/ae821819bef4404294bc870a1597cf64, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/5233ec518803433bad5c9d0efa7ea40f, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/2eb1ff62a6064687a8818a54b4ce106b, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/59bac3d9b1b34d1188aba5c0152c0f59, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/2762c960aec24d86bfb7c0a075a48a78, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/15792f21d70c4057826d9060c2ec1bb6, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/511e30f9f46d4d86aae5ae1b9d22e146, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/8e8e5ad0d2b44d9bb3977c9daf67108d, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/95eee51867de4585b5da7fe182f100c3, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/49b6d93777064bd389c5e304363837e5, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/3a6b8198c56047fa9f55bce050f5eb83, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/d12bcdc968304759b8bd4a6baa702ec0, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/7257ed6fb8374f3ab789ea9c21512964, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/67fd81ac447e481bbf4d24eff5e87bda, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/ca13e72b97cc4e238f1c3aefc01f007f, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/1a5c757d001c44d1bd1cb858c34426e7, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/a569198ba4d445669ef81b235c14ff57, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/8dde77a6f07e4cd4a749148263e65e67, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/2a673f25d2624aa09106d31d7696311e, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/0458b8415c624f80a1a0f4a75e8d66a7, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/309858ab27fc48369f1d6a4dd7bd78f6, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/e4ea900cb7b249c690f5dabded681cf9, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/548fac84597544b7aa3353b95607a054, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/fd79b6156cd94489820ffcc7eada3452, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/318be65f80634fa0b4defcc935446002, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/302828fd5e1348da88e4f36a38ae9fdb, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/f9b00f77845a4984ba7f60304d5ec246, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/db41dc240c404d13b7e942b99c06d4a7, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/10f4691964204c6888d0f9d66c5d781c] to archive 2023-04-19 20:56:03,551 DEBUG [StoreCloser-TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-19 20:56:03,553 DEBUG [StoreCloser-TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/593cad6360284a51a3288515b09116e0 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/593cad6360284a51a3288515b09116e0 2023-04-19 20:56:03,554 DEBUG [StoreCloser-TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/0e6df69316b3476995b3bbd854a55c3f to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/0e6df69316b3476995b3bbd854a55c3f 2023-04-19 20:56:03,554 DEBUG [StoreCloser-TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/6508ddb2cff04e129cbb9f90ca8b10ef to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/6508ddb2cff04e129cbb9f90ca8b10ef 2023-04-19 20:56:03,555 DEBUG [StoreCloser-TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/fc4621db25f14d989e8f3460cfd0d42f to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/fc4621db25f14d989e8f3460cfd0d42f 2023-04-19 20:56:03,556 DEBUG [StoreCloser-TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/f19b87eaae6941969bb821a857ca9dc8 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/f19b87eaae6941969bb821a857ca9dc8 2023-04-19 20:56:03,557 DEBUG [StoreCloser-TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/115eea192f7b4b21a872c8f0c75a8813 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/115eea192f7b4b21a872c8f0c75a8813 2023-04-19 20:56:03,557 DEBUG [StoreCloser-TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/b60ee764ac8e437091ed5a69bfaa3b13 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/b60ee764ac8e437091ed5a69bfaa3b13 2023-04-19 20:56:03,558 DEBUG [StoreCloser-TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/e87262c540ef4372b3ec73f498a60b28 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/e87262c540ef4372b3ec73f498a60b28 2023-04-19 20:56:03,559 DEBUG [StoreCloser-TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/bdea0565ed934118b46236767cc2487b to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/bdea0565ed934118b46236767cc2487b 2023-04-19 20:56:03,560 DEBUG [StoreCloser-TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/9ea01a31f7cb415ebbef0aa052832786 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/9ea01a31f7cb415ebbef0aa052832786 2023-04-19 20:56:03,560 DEBUG [StoreCloser-TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/9a91bef8a8544db6a38a37d8b1eb6d58 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/9a91bef8a8544db6a38a37d8b1eb6d58 2023-04-19 20:56:03,561 DEBUG [StoreCloser-TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/3ffc60b4c21d4ccd8c01900796cac8cf to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/3ffc60b4c21d4ccd8c01900796cac8cf 2023-04-19 20:56:03,562 DEBUG [StoreCloser-TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/0f1c5c4ed0b14221b9e65af5aaa0e473 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/0f1c5c4ed0b14221b9e65af5aaa0e473 2023-04-19 20:56:03,562 DEBUG [StoreCloser-TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/15edd9481e0c4eb4a4e6176a7acb7ae8 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/15edd9481e0c4eb4a4e6176a7acb7ae8 2023-04-19 20:56:03,563 DEBUG [StoreCloser-TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/66568d2c0e694ec59e2758f088f94f96 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/66568d2c0e694ec59e2758f088f94f96 2023-04-19 20:56:03,564 DEBUG [StoreCloser-TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/785eec2bf2554d30b5c16f9a25379568 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/785eec2bf2554d30b5c16f9a25379568 2023-04-19 20:56:03,565 DEBUG [StoreCloser-TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/46d649f1d4c64cbbaad84dd81d645bac to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/46d649f1d4c64cbbaad84dd81d645bac 2023-04-19 20:56:03,565 DEBUG [StoreCloser-TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/83f1252ec5f44d1585561d734723ea22 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/83f1252ec5f44d1585561d734723ea22 2023-04-19 20:56:03,566 DEBUG [StoreCloser-TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/476f2dffd0a0400d8b8e9e7891445841 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/476f2dffd0a0400d8b8e9e7891445841 2023-04-19 20:56:03,567 DEBUG [StoreCloser-TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/3d713cd6cad44bf8909494a274ee3256 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/3d713cd6cad44bf8909494a274ee3256 2023-04-19 20:56:03,568 DEBUG [StoreCloser-TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/1b0d4fd1deee4abab035f01ee9d1d0c0 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/1b0d4fd1deee4abab035f01ee9d1d0c0 2023-04-19 20:56:03,568 DEBUG [StoreCloser-TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/f3aed609eeee412dbad9a1322a48bb8f to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/f3aed609eeee412dbad9a1322a48bb8f 2023-04-19 20:56:03,569 DEBUG [StoreCloser-TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/ae821819bef4404294bc870a1597cf64 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/ae821819bef4404294bc870a1597cf64 2023-04-19 20:56:03,570 DEBUG [StoreCloser-TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/5233ec518803433bad5c9d0efa7ea40f to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/5233ec518803433bad5c9d0efa7ea40f 2023-04-19 20:56:03,571 DEBUG [StoreCloser-TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/2eb1ff62a6064687a8818a54b4ce106b to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/2eb1ff62a6064687a8818a54b4ce106b 2023-04-19 20:56:03,571 DEBUG [StoreCloser-TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/59bac3d9b1b34d1188aba5c0152c0f59 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/59bac3d9b1b34d1188aba5c0152c0f59 2023-04-19 20:56:03,572 DEBUG [StoreCloser-TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/2762c960aec24d86bfb7c0a075a48a78 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/2762c960aec24d86bfb7c0a075a48a78 2023-04-19 20:56:03,573 DEBUG [StoreCloser-TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/15792f21d70c4057826d9060c2ec1bb6 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/15792f21d70c4057826d9060c2ec1bb6 2023-04-19 20:56:03,573 DEBUG [StoreCloser-TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/511e30f9f46d4d86aae5ae1b9d22e146 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/511e30f9f46d4d86aae5ae1b9d22e146 2023-04-19 20:56:03,575 DEBUG [StoreCloser-TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/8e8e5ad0d2b44d9bb3977c9daf67108d to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/8e8e5ad0d2b44d9bb3977c9daf67108d 2023-04-19 20:56:03,576 DEBUG [StoreCloser-TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/95eee51867de4585b5da7fe182f100c3 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/95eee51867de4585b5da7fe182f100c3 2023-04-19 20:56:03,577 DEBUG [StoreCloser-TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/49b6d93777064bd389c5e304363837e5 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/49b6d93777064bd389c5e304363837e5 2023-04-19 20:56:03,578 DEBUG [StoreCloser-TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/3a6b8198c56047fa9f55bce050f5eb83 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/3a6b8198c56047fa9f55bce050f5eb83 2023-04-19 20:56:03,579 DEBUG [StoreCloser-TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/d12bcdc968304759b8bd4a6baa702ec0 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/d12bcdc968304759b8bd4a6baa702ec0 2023-04-19 20:56:03,580 DEBUG [StoreCloser-TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/7257ed6fb8374f3ab789ea9c21512964 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/7257ed6fb8374f3ab789ea9c21512964 2023-04-19 20:56:03,581 DEBUG [StoreCloser-TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/67fd81ac447e481bbf4d24eff5e87bda to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/67fd81ac447e481bbf4d24eff5e87bda 2023-04-19 20:56:03,582 DEBUG [StoreCloser-TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/ca13e72b97cc4e238f1c3aefc01f007f to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/ca13e72b97cc4e238f1c3aefc01f007f 2023-04-19 20:56:03,583 DEBUG [StoreCloser-TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/1a5c757d001c44d1bd1cb858c34426e7 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/1a5c757d001c44d1bd1cb858c34426e7 2023-04-19 20:56:03,584 DEBUG [StoreCloser-TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/a569198ba4d445669ef81b235c14ff57 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/a569198ba4d445669ef81b235c14ff57 2023-04-19 20:56:03,585 DEBUG [StoreCloser-TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/8dde77a6f07e4cd4a749148263e65e67 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/8dde77a6f07e4cd4a749148263e65e67 2023-04-19 20:56:03,586 DEBUG [StoreCloser-TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/2a673f25d2624aa09106d31d7696311e to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/2a673f25d2624aa09106d31d7696311e 2023-04-19 20:56:03,587 DEBUG [StoreCloser-TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/0458b8415c624f80a1a0f4a75e8d66a7 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/0458b8415c624f80a1a0f4a75e8d66a7 2023-04-19 20:56:03,588 DEBUG [StoreCloser-TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/309858ab27fc48369f1d6a4dd7bd78f6 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/309858ab27fc48369f1d6a4dd7bd78f6 2023-04-19 20:56:03,589 DEBUG [StoreCloser-TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/e4ea900cb7b249c690f5dabded681cf9 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/e4ea900cb7b249c690f5dabded681cf9 2023-04-19 20:56:03,590 DEBUG [StoreCloser-TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/548fac84597544b7aa3353b95607a054 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/548fac84597544b7aa3353b95607a054 2023-04-19 20:56:03,591 DEBUG [StoreCloser-TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/fd79b6156cd94489820ffcc7eada3452 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/fd79b6156cd94489820ffcc7eada3452 2023-04-19 20:56:03,592 DEBUG [StoreCloser-TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/318be65f80634fa0b4defcc935446002 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/318be65f80634fa0b4defcc935446002 2023-04-19 20:56:03,593 DEBUG [StoreCloser-TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/302828fd5e1348da88e4f36a38ae9fdb to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/302828fd5e1348da88e4f36a38ae9fdb 2023-04-19 20:56:03,594 DEBUG [StoreCloser-TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/f9b00f77845a4984ba7f60304d5ec246 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/f9b00f77845a4984ba7f60304d5ec246 2023-04-19 20:56:03,595 DEBUG [StoreCloser-TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/db41dc240c404d13b7e942b99c06d4a7 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/db41dc240c404d13b7e942b99c06d4a7 2023-04-19 20:56:03,596 DEBUG [StoreCloser-TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/10f4691964204c6888d0f9d66c5d781c to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/10f4691964204c6888d0f9d66c5d781c 2023-04-19 20:56:03,601 DEBUG [StoreCloser-TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/c20e704eaa9047948b65bb0521962ae2, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/27dae2239ac84084b6d41bdb3cd6a0aa, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/57970f5ab2d24c53b63b790d1a4dcd79, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/edda786758cb4b6c901a76f7c035e065, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/7f0298a614364121a886b61a788cffc5, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/54c4398026344169970b653f9f5362e3, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/7e5292be149e42b2a9fb18706abe24d7, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/a86e2147024c4edfad3b8b32a904ebbe, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/e763ddb01c92427db22e8f6a441d3bcf, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/01c9a67084494cec85710db4418b8cc3, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/0e83b450afbb46e79b96d9df35f0f0d3, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/9acd5806f2cc440f823b467a13248046, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/e6c2e4affed44fd48ab9261a0706d2f4, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/afe0cadbf1ff4a73a34b4194726fad5a, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/472e8d25dc514814b624e1d57ed53ec6, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/a3330176b169407a981620c68915d582, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/2fdbdee3b0c04d0f8490ceb791ca32ab, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/b25198594675479583e0b463c8587b1c, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/22e2a025b60b4298bcb1fb60083eb171, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/9a2493853ea342f7b2bff95de1f09376, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/19dd7570fe064ae1aa15f42c83abebe2, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/f5f5527ed09f4697b20a7cce4ead1353, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/45f2ee1ae15e413088ef7d40130256b8, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/766ca41fef7a4f9fa07fd0f5deac3107, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/5aef647ab52c412fb92806e34a67b515, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/edde5ed5b4a04bebb9a4de18010ec55b, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/4ac0391327d644b2b7c688dbbce1f03e, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/192809bb0249425883b8a36031c86f31, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/9739c501febe47e99417caa65c62a5fb, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/7ac6539cfbcd474181bcd610158a054f, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/8550f0c81b7046baacb90f80db34e2f3, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/2fd9720af527423698fa9ed7a9d2299c, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/635db6a1d809458d80e73e3b897cf07b, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/824013430a36411f82d84cc69eb916ac, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/619ac4ba267946ecbd77864f8739f7ab, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/e5066d74e0d74d908fa018f82896a5ea, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/b5a024111ffd4368a7b632cfbab91ac9, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/51eddd72661142fb8a4cc4f7e4ba2ffe, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/fa8245b01998409cb6472803c40fbdee, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/4891ddad549e46d5bf87e3bef789d199, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/1b2a0ef114584e59811edfe9b80be235, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/6858cf98df92400db5f9eea4956bd8a7, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/38e1cc4375034e13b2652139e727fdf5, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/f300d42c7c5048619332277a7d304dae, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/2d5bfb8ba7604e86a0ee3f058a27f2ca, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/b2cf2936dbe54253b08855f45730c8be, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/7b941f46c179455e9ed81fe241ff0761, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/717b44d64f824940b5bb5a04ca662a12, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/533a96acbb4f473fb1601db968733253, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/ac03180f8ee24999a82b039fab20977c, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/473c92cb23c342eb84038b54d357ef30] to archive 2023-04-19 20:56:03,601 DEBUG [StoreCloser-TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-19 20:56:03,603 DEBUG [StoreCloser-TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/c20e704eaa9047948b65bb0521962ae2 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/c20e704eaa9047948b65bb0521962ae2 2023-04-19 20:56:03,604 DEBUG [StoreCloser-TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/27dae2239ac84084b6d41bdb3cd6a0aa to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/27dae2239ac84084b6d41bdb3cd6a0aa 2023-04-19 20:56:03,605 DEBUG [StoreCloser-TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/57970f5ab2d24c53b63b790d1a4dcd79 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/57970f5ab2d24c53b63b790d1a4dcd79 2023-04-19 20:56:03,606 DEBUG [StoreCloser-TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/edda786758cb4b6c901a76f7c035e065 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/edda786758cb4b6c901a76f7c035e065 2023-04-19 20:56:03,607 DEBUG [StoreCloser-TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/7f0298a614364121a886b61a788cffc5 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/7f0298a614364121a886b61a788cffc5 2023-04-19 20:56:03,608 DEBUG [StoreCloser-TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/54c4398026344169970b653f9f5362e3 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/54c4398026344169970b653f9f5362e3 2023-04-19 20:56:03,608 DEBUG [StoreCloser-TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/7e5292be149e42b2a9fb18706abe24d7 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/7e5292be149e42b2a9fb18706abe24d7 2023-04-19 20:56:03,609 DEBUG [StoreCloser-TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/a86e2147024c4edfad3b8b32a904ebbe to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/a86e2147024c4edfad3b8b32a904ebbe 2023-04-19 20:56:03,610 DEBUG [StoreCloser-TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/e763ddb01c92427db22e8f6a441d3bcf to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/e763ddb01c92427db22e8f6a441d3bcf 2023-04-19 20:56:03,611 DEBUG [StoreCloser-TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/01c9a67084494cec85710db4418b8cc3 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/01c9a67084494cec85710db4418b8cc3 2023-04-19 20:56:03,612 DEBUG [StoreCloser-TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/0e83b450afbb46e79b96d9df35f0f0d3 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/0e83b450afbb46e79b96d9df35f0f0d3 2023-04-19 20:56:03,612 DEBUG [StoreCloser-TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/9acd5806f2cc440f823b467a13248046 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/9acd5806f2cc440f823b467a13248046 2023-04-19 20:56:03,613 DEBUG [StoreCloser-TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/e6c2e4affed44fd48ab9261a0706d2f4 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/e6c2e4affed44fd48ab9261a0706d2f4 2023-04-19 20:56:03,614 DEBUG [StoreCloser-TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/afe0cadbf1ff4a73a34b4194726fad5a to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/afe0cadbf1ff4a73a34b4194726fad5a 2023-04-19 20:56:03,615 DEBUG [StoreCloser-TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/472e8d25dc514814b624e1d57ed53ec6 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/472e8d25dc514814b624e1d57ed53ec6 2023-04-19 20:56:03,616 DEBUG [StoreCloser-TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/a3330176b169407a981620c68915d582 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/a3330176b169407a981620c68915d582 2023-04-19 20:56:03,616 DEBUG [StoreCloser-TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/2fdbdee3b0c04d0f8490ceb791ca32ab to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/2fdbdee3b0c04d0f8490ceb791ca32ab 2023-04-19 20:56:03,617 DEBUG [StoreCloser-TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/b25198594675479583e0b463c8587b1c to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/b25198594675479583e0b463c8587b1c 2023-04-19 20:56:03,618 DEBUG [StoreCloser-TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/22e2a025b60b4298bcb1fb60083eb171 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/22e2a025b60b4298bcb1fb60083eb171 2023-04-19 20:56:03,619 DEBUG [StoreCloser-TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/9a2493853ea342f7b2bff95de1f09376 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/9a2493853ea342f7b2bff95de1f09376 2023-04-19 20:56:03,619 DEBUG [StoreCloser-TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/19dd7570fe064ae1aa15f42c83abebe2 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/19dd7570fe064ae1aa15f42c83abebe2 2023-04-19 20:56:03,621 DEBUG [StoreCloser-TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/f5f5527ed09f4697b20a7cce4ead1353 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/f5f5527ed09f4697b20a7cce4ead1353 2023-04-19 20:56:03,622 DEBUG [StoreCloser-TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/45f2ee1ae15e413088ef7d40130256b8 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/45f2ee1ae15e413088ef7d40130256b8 2023-04-19 20:56:03,622 DEBUG [StoreCloser-TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/766ca41fef7a4f9fa07fd0f5deac3107 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/766ca41fef7a4f9fa07fd0f5deac3107 2023-04-19 20:56:03,623 DEBUG [StoreCloser-TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/5aef647ab52c412fb92806e34a67b515 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/5aef647ab52c412fb92806e34a67b515 2023-04-19 20:56:03,624 DEBUG [StoreCloser-TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/edde5ed5b4a04bebb9a4de18010ec55b to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/edde5ed5b4a04bebb9a4de18010ec55b 2023-04-19 20:56:03,625 DEBUG [StoreCloser-TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/4ac0391327d644b2b7c688dbbce1f03e to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/4ac0391327d644b2b7c688dbbce1f03e 2023-04-19 20:56:03,626 DEBUG [StoreCloser-TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/192809bb0249425883b8a36031c86f31 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/192809bb0249425883b8a36031c86f31 2023-04-19 20:56:03,627 DEBUG [StoreCloser-TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/9739c501febe47e99417caa65c62a5fb to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/9739c501febe47e99417caa65c62a5fb 2023-04-19 20:56:03,628 DEBUG [StoreCloser-TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/7ac6539cfbcd474181bcd610158a054f to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/7ac6539cfbcd474181bcd610158a054f 2023-04-19 20:56:03,629 DEBUG [StoreCloser-TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/8550f0c81b7046baacb90f80db34e2f3 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/8550f0c81b7046baacb90f80db34e2f3 2023-04-19 20:56:03,630 DEBUG [StoreCloser-TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/2fd9720af527423698fa9ed7a9d2299c to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/2fd9720af527423698fa9ed7a9d2299c 2023-04-19 20:56:03,630 DEBUG [StoreCloser-TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/635db6a1d809458d80e73e3b897cf07b to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/635db6a1d809458d80e73e3b897cf07b 2023-04-19 20:56:03,631 DEBUG [StoreCloser-TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/824013430a36411f82d84cc69eb916ac to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/824013430a36411f82d84cc69eb916ac 2023-04-19 20:56:03,632 DEBUG [StoreCloser-TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/619ac4ba267946ecbd77864f8739f7ab to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/619ac4ba267946ecbd77864f8739f7ab 2023-04-19 20:56:03,633 DEBUG [StoreCloser-TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/e5066d74e0d74d908fa018f82896a5ea to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/e5066d74e0d74d908fa018f82896a5ea 2023-04-19 20:56:03,634 DEBUG [StoreCloser-TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/b5a024111ffd4368a7b632cfbab91ac9 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/b5a024111ffd4368a7b632cfbab91ac9 2023-04-19 20:56:03,634 DEBUG [StoreCloser-TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/51eddd72661142fb8a4cc4f7e4ba2ffe to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/51eddd72661142fb8a4cc4f7e4ba2ffe 2023-04-19 20:56:03,635 DEBUG [StoreCloser-TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/fa8245b01998409cb6472803c40fbdee to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/fa8245b01998409cb6472803c40fbdee 2023-04-19 20:56:03,637 DEBUG [StoreCloser-TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/4891ddad549e46d5bf87e3bef789d199 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/4891ddad549e46d5bf87e3bef789d199 2023-04-19 20:56:03,637 DEBUG [StoreCloser-TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/1b2a0ef114584e59811edfe9b80be235 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/1b2a0ef114584e59811edfe9b80be235 2023-04-19 20:56:03,638 DEBUG [StoreCloser-TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/6858cf98df92400db5f9eea4956bd8a7 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/6858cf98df92400db5f9eea4956bd8a7 2023-04-19 20:56:03,639 DEBUG [StoreCloser-TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/38e1cc4375034e13b2652139e727fdf5 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/38e1cc4375034e13b2652139e727fdf5 2023-04-19 20:56:03,640 DEBUG [StoreCloser-TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/f300d42c7c5048619332277a7d304dae to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/f300d42c7c5048619332277a7d304dae 2023-04-19 20:56:03,641 DEBUG [StoreCloser-TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/2d5bfb8ba7604e86a0ee3f058a27f2ca to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/2d5bfb8ba7604e86a0ee3f058a27f2ca 2023-04-19 20:56:03,642 DEBUG [StoreCloser-TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/b2cf2936dbe54253b08855f45730c8be to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/b2cf2936dbe54253b08855f45730c8be 2023-04-19 20:56:03,643 DEBUG [StoreCloser-TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/7b941f46c179455e9ed81fe241ff0761 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/7b941f46c179455e9ed81fe241ff0761 2023-04-19 20:56:03,644 DEBUG [StoreCloser-TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/717b44d64f824940b5bb5a04ca662a12 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/717b44d64f824940b5bb5a04ca662a12 2023-04-19 20:56:03,645 DEBUG [StoreCloser-TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/533a96acbb4f473fb1601db968733253 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/533a96acbb4f473fb1601db968733253 2023-04-19 20:56:03,646 DEBUG [StoreCloser-TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/ac03180f8ee24999a82b039fab20977c to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/ac03180f8ee24999a82b039fab20977c 2023-04-19 20:56:03,646 DEBUG [StoreCloser-TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/473c92cb23c342eb84038b54d357ef30 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/473c92cb23c342eb84038b54d357ef30 2023-04-19 20:56:03,650 DEBUG [StoreCloser-TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/b6ecd3d5a379435b990ecc11fae75469, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/131e9ee789ff4ccc99ca836091dd6321, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/5b670c0b836c4992894eb3acc1ab53d1, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/f2d9e2014e9040f5a8c9922c97ba5f69, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/a458a37c08fa470ea7b666b009d5a409, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/61468d6ab0ae4b31acd445825ed63fc8, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/9da508b6c036409b8668ece4023d04f4, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/b616a1072bfb42f4928243d32fbd9d65, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/86989a3cc20441d896ad38e61fb52150, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/b9e3de9a3ac541f3be7ed560f00da299, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/add62c469c004655a1cbf2ad2844a86b, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/e92c9c81e9b04913b595b2510fbf09d6, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/1c4ea4067db14336912112f75758f28c, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/aa28b17809d94b3a996f3c168c7d0799, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/e3b17b126f24483083b3bd2d89935385, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/c543956a42754d7aa0512223ea0af147, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/9f1c8421794945c6ac876bb703cffcfb, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/717cf7885aff481499668309cb0affbc, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/eb59f8cd86e84e779614aeb9b2dad22e, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/8aab4a235bb34537849c7f92acaa9ac0, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/537489db7e744a4b89644c6fb462bc3c, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/6bf6cc3d591d4b06ab6918418e3acdc6, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/9d6ecbee77b84887bde1852370de6152, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/bc6301135b994e2c9b159011972897b8, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/47fd38401d6e4e24bdc64b779e839c09, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/fefd7d901c534a8a8ac20a1ac0bbbf70, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/e823084c3f004185a2ef1ced67cafc9a, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/e4141273854e4a3a9a931efc95e8a83d, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/e7c8e6b0fa934031a1395dcdc0c697e6, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/f18549fccb9f478c925b497a8c3a6513, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/3e96a02b88904cb09ee3b3488a81a159, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/fedbba22989346bd9f9d053a55b21d12, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/50ebf1cf050047ba9d73391a1169e9ee, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/8b0eefd0196a45b8b810ca16eee6bb25, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/d2e9d238194d44daa005f1e915327101, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/069b5cab189647e68c3689b0b13ac09b, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/2c628db85cf04906bbd04c1a65e4b5fc, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/5f9200ec194e4b9ea98ee198625feab6, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/08e34fe8e6c84e8ea85edf580f24ddcc, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/631c453b03984a5181f72a41b46d0fca, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/ab2fb1ab7b024df587d4d03f94257985, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/b634153cab4148388e9e607cbbdcdf58, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/1864efc9f6194856b7fcf77ec52cbab8, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/5397c2fbad0249d4a559380e55097d25, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/ff31eae303cc4f7f83a58ec6b50c69ea, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/4462551c90984908a8869f965accf87b, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/2dc58269c48d45a0a10d64830673f6d1, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/e9451d05b033439bba2b444d75b7dd54, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/d5086dcd74f0450497e1768e1b0c6afd] to archive 2023-04-19 20:56:03,651 DEBUG [StoreCloser-TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-19 20:56:03,652 DEBUG [StoreCloser-TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/b6ecd3d5a379435b990ecc11fae75469 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/b6ecd3d5a379435b990ecc11fae75469 2023-04-19 20:56:03,653 DEBUG [StoreCloser-TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/131e9ee789ff4ccc99ca836091dd6321 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/131e9ee789ff4ccc99ca836091dd6321 2023-04-19 20:56:03,653 DEBUG [StoreCloser-TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/5b670c0b836c4992894eb3acc1ab53d1 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/5b670c0b836c4992894eb3acc1ab53d1 2023-04-19 20:56:03,654 DEBUG [StoreCloser-TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/f2d9e2014e9040f5a8c9922c97ba5f69 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/f2d9e2014e9040f5a8c9922c97ba5f69 2023-04-19 20:56:03,655 DEBUG [StoreCloser-TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/a458a37c08fa470ea7b666b009d5a409 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/a458a37c08fa470ea7b666b009d5a409 2023-04-19 20:56:03,656 DEBUG [StoreCloser-TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/61468d6ab0ae4b31acd445825ed63fc8 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/61468d6ab0ae4b31acd445825ed63fc8 2023-04-19 20:56:03,656 DEBUG [StoreCloser-TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/9da508b6c036409b8668ece4023d04f4 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/9da508b6c036409b8668ece4023d04f4 2023-04-19 20:56:03,657 DEBUG [StoreCloser-TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/b616a1072bfb42f4928243d32fbd9d65 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/b616a1072bfb42f4928243d32fbd9d65 2023-04-19 20:56:03,658 DEBUG [StoreCloser-TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/86989a3cc20441d896ad38e61fb52150 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/86989a3cc20441d896ad38e61fb52150 2023-04-19 20:56:03,659 DEBUG [StoreCloser-TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/b9e3de9a3ac541f3be7ed560f00da299 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/b9e3de9a3ac541f3be7ed560f00da299 2023-04-19 20:56:03,659 DEBUG [StoreCloser-TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/add62c469c004655a1cbf2ad2844a86b to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/add62c469c004655a1cbf2ad2844a86b 2023-04-19 20:56:03,660 DEBUG [StoreCloser-TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/e92c9c81e9b04913b595b2510fbf09d6 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/e92c9c81e9b04913b595b2510fbf09d6 2023-04-19 20:56:03,661 DEBUG [StoreCloser-TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/1c4ea4067db14336912112f75758f28c to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/1c4ea4067db14336912112f75758f28c 2023-04-19 20:56:03,662 DEBUG [StoreCloser-TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/aa28b17809d94b3a996f3c168c7d0799 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/aa28b17809d94b3a996f3c168c7d0799 2023-04-19 20:56:03,662 DEBUG [StoreCloser-TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/e3b17b126f24483083b3bd2d89935385 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/e3b17b126f24483083b3bd2d89935385 2023-04-19 20:56:03,663 DEBUG [StoreCloser-TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/c543956a42754d7aa0512223ea0af147 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/c543956a42754d7aa0512223ea0af147 2023-04-19 20:56:03,664 DEBUG [StoreCloser-TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/9f1c8421794945c6ac876bb703cffcfb to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/9f1c8421794945c6ac876bb703cffcfb 2023-04-19 20:56:03,664 DEBUG [StoreCloser-TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/717cf7885aff481499668309cb0affbc to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/717cf7885aff481499668309cb0affbc 2023-04-19 20:56:03,665 DEBUG [StoreCloser-TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/eb59f8cd86e84e779614aeb9b2dad22e to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/eb59f8cd86e84e779614aeb9b2dad22e 2023-04-19 20:56:03,666 DEBUG [StoreCloser-TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/8aab4a235bb34537849c7f92acaa9ac0 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/8aab4a235bb34537849c7f92acaa9ac0 2023-04-19 20:56:03,667 DEBUG [StoreCloser-TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/537489db7e744a4b89644c6fb462bc3c to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/537489db7e744a4b89644c6fb462bc3c 2023-04-19 20:56:03,667 DEBUG [StoreCloser-TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/6bf6cc3d591d4b06ab6918418e3acdc6 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/6bf6cc3d591d4b06ab6918418e3acdc6 2023-04-19 20:56:03,668 DEBUG [StoreCloser-TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/9d6ecbee77b84887bde1852370de6152 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/9d6ecbee77b84887bde1852370de6152 2023-04-19 20:56:03,669 DEBUG [StoreCloser-TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/bc6301135b994e2c9b159011972897b8 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/bc6301135b994e2c9b159011972897b8 2023-04-19 20:56:03,670 DEBUG [StoreCloser-TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/47fd38401d6e4e24bdc64b779e839c09 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/47fd38401d6e4e24bdc64b779e839c09 2023-04-19 20:56:03,670 DEBUG [StoreCloser-TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/fefd7d901c534a8a8ac20a1ac0bbbf70 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/fefd7d901c534a8a8ac20a1ac0bbbf70 2023-04-19 20:56:03,671 DEBUG [StoreCloser-TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/e823084c3f004185a2ef1ced67cafc9a to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/e823084c3f004185a2ef1ced67cafc9a 2023-04-19 20:56:03,672 DEBUG [StoreCloser-TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/e4141273854e4a3a9a931efc95e8a83d to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/e4141273854e4a3a9a931efc95e8a83d 2023-04-19 20:56:03,673 DEBUG [StoreCloser-TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/e7c8e6b0fa934031a1395dcdc0c697e6 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/e7c8e6b0fa934031a1395dcdc0c697e6 2023-04-19 20:56:03,673 DEBUG [StoreCloser-TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/f18549fccb9f478c925b497a8c3a6513 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/f18549fccb9f478c925b497a8c3a6513 2023-04-19 20:56:03,674 DEBUG [StoreCloser-TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/3e96a02b88904cb09ee3b3488a81a159 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/3e96a02b88904cb09ee3b3488a81a159 2023-04-19 20:56:03,675 DEBUG [StoreCloser-TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/fedbba22989346bd9f9d053a55b21d12 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/fedbba22989346bd9f9d053a55b21d12 2023-04-19 20:56:03,676 DEBUG [StoreCloser-TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/50ebf1cf050047ba9d73391a1169e9ee to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/50ebf1cf050047ba9d73391a1169e9ee 2023-04-19 20:56:03,676 DEBUG [StoreCloser-TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/8b0eefd0196a45b8b810ca16eee6bb25 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/8b0eefd0196a45b8b810ca16eee6bb25 2023-04-19 20:56:03,677 DEBUG [StoreCloser-TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/d2e9d238194d44daa005f1e915327101 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/d2e9d238194d44daa005f1e915327101 2023-04-19 20:56:03,678 DEBUG [StoreCloser-TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/069b5cab189647e68c3689b0b13ac09b to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/069b5cab189647e68c3689b0b13ac09b 2023-04-19 20:56:03,679 DEBUG [StoreCloser-TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/2c628db85cf04906bbd04c1a65e4b5fc to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/2c628db85cf04906bbd04c1a65e4b5fc 2023-04-19 20:56:03,680 DEBUG [StoreCloser-TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/5f9200ec194e4b9ea98ee198625feab6 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/5f9200ec194e4b9ea98ee198625feab6 2023-04-19 20:56:03,681 DEBUG [StoreCloser-TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/08e34fe8e6c84e8ea85edf580f24ddcc to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/08e34fe8e6c84e8ea85edf580f24ddcc 2023-04-19 20:56:03,681 DEBUG [StoreCloser-TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/631c453b03984a5181f72a41b46d0fca to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/631c453b03984a5181f72a41b46d0fca 2023-04-19 20:56:03,682 DEBUG [StoreCloser-TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/ab2fb1ab7b024df587d4d03f94257985 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/ab2fb1ab7b024df587d4d03f94257985 2023-04-19 20:56:03,683 DEBUG [StoreCloser-TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/b634153cab4148388e9e607cbbdcdf58 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/b634153cab4148388e9e607cbbdcdf58 2023-04-19 20:56:03,684 DEBUG [StoreCloser-TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/1864efc9f6194856b7fcf77ec52cbab8 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/1864efc9f6194856b7fcf77ec52cbab8 2023-04-19 20:56:03,684 DEBUG [StoreCloser-TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/5397c2fbad0249d4a559380e55097d25 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/5397c2fbad0249d4a559380e55097d25 2023-04-19 20:56:03,685 DEBUG [StoreCloser-TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/ff31eae303cc4f7f83a58ec6b50c69ea to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/ff31eae303cc4f7f83a58ec6b50c69ea 2023-04-19 20:56:03,686 DEBUG [StoreCloser-TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/4462551c90984908a8869f965accf87b to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/4462551c90984908a8869f965accf87b 2023-04-19 20:56:03,686 DEBUG [StoreCloser-TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/2dc58269c48d45a0a10d64830673f6d1 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/2dc58269c48d45a0a10d64830673f6d1 2023-04-19 20:56:03,687 DEBUG [StoreCloser-TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/e9451d05b033439bba2b444d75b7dd54 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/e9451d05b033439bba2b444d75b7dd54 2023-04-19 20:56:03,688 DEBUG [StoreCloser-TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/d5086dcd74f0450497e1768e1b0c6afd to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/d5086dcd74f0450497e1768e1b0c6afd 2023-04-19 20:56:03,691 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/recovered.edits/1131.seqid, newMaxSeqId=1131, maxSeqId=1 2023-04-19 20:56:03,692 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d. 2023-04-19 20:56:03,693 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 4393734ae917e967cf2b5b4b3b44ed0d: 2023-04-19 20:56:03,694 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 4393734ae917e967cf2b5b4b3b44ed0d 2023-04-19 20:56:03,694 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=4393734ae917e967cf2b5b4b3b44ed0d, regionState=CLOSED 2023-04-19 20:56:03,694 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681937763694"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681937763694"}]},"ts":"1681937763694"} 2023-04-19 20:56:03,696 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=52, resume processing ppid=51 2023-04-19 20:56:03,696 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=52, ppid=51, state=SUCCESS; CloseRegionProcedure 4393734ae917e967cf2b5b4b3b44ed0d, server=jenkins-hbase4.apache.org,35247,1681937616971 in 1.5660 sec 2023-04-19 20:56:03,697 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=51, resume processing ppid=50 2023-04-19 20:56:03,697 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=51, ppid=50, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=4393734ae917e967cf2b5b4b3b44ed0d, UNASSIGN in 1.5690 sec 2023-04-19 20:56:03,698 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681937763697"}]},"ts":"1681937763697"} 2023-04-19 20:56:03,698 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-19 20:56:03,700 INFO [PEWorker-4] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-19 20:56:03,701 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=50, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.5800 sec 2023-04-19 20:56:04,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-19 20:56:04,228 INFO [Listener at localhost/34893] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 50 completed 2023-04-19 20:56:04,229 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-19 20:56:04,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure2.ProcedureExecutor(1029): Stored pid=53, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-19 20:56:04,231 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=53, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-19 20:56:04,232 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=53, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-19 20:56:04,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1227): Checking to see if procedure is done pid=53 2023-04-19 20:56:04,236 DEBUG [HFileArchiver-11] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/.tmp/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d 2023-04-19 20:56:04,238 DEBUG [HFileArchiver-11] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/.tmp/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A, FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/.tmp/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B, FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/.tmp/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C, FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/.tmp/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/recovered.edits] 2023-04-19 20:56:04,240 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/.tmp/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/9f18b0eb5747402fa6ee1b1a6064262b to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/9f18b0eb5747402fa6ee1b1a6064262b 2023-04-19 20:56:04,241 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/.tmp/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/a6e8d7d34347445cbc42b90de7a98cb7 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/a6e8d7d34347445cbc42b90de7a98cb7 2023-04-19 20:56:04,242 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/.tmp/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/b695deeeb77b4b9caadb1846dc79dd44 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/A/b695deeeb77b4b9caadb1846dc79dd44 2023-04-19 20:56:04,244 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/.tmp/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/65e068f4295547ad99d13aee864dd010 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/65e068f4295547ad99d13aee864dd010 2023-04-19 20:56:04,245 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/.tmp/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/bf4a2cc56ba9479fa8cbdbc30b143f78 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/bf4a2cc56ba9479fa8cbdbc30b143f78 2023-04-19 20:56:04,245 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/.tmp/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/fb50241348e240d18b98afb69ce56b21 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/B/fb50241348e240d18b98afb69ce56b21 2023-04-19 20:56:04,247 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/.tmp/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/23d714a08c8f41518adfdf2581ad1285 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/23d714a08c8f41518adfdf2581ad1285 2023-04-19 20:56:04,248 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/.tmp/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/332aaadde0174cc2a72993ce24de6c9a to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/332aaadde0174cc2a72993ce24de6c9a 2023-04-19 20:56:04,249 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/.tmp/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/52688eb791c445f2b2868fbcf00586c9 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/C/52688eb791c445f2b2868fbcf00586c9 2023-04-19 20:56:04,251 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/.tmp/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/recovered.edits/1131.seqid to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d/recovered.edits/1131.seqid 2023-04-19 20:56:04,251 DEBUG [HFileArchiver-11] backup.HFileArchiver(596): Deleted hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/.tmp/data/default/TestAcidGuarantees/4393734ae917e967cf2b5b4b3b44ed0d 2023-04-19 20:56:04,251 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-19 20:56:04,253 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=53, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-19 20:56:04,254 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-19 20:56:04,255 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-19 20:56:04,256 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=53, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-19 20:56:04,256 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-19 20:56:04,256 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1681937764256"}]},"ts":"9223372036854775807"} 2023-04-19 20:56:04,258 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-19 20:56:04,258 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 4393734ae917e967cf2b5b4b3b44ed0d, NAME => 'TestAcidGuarantees,,1681937739023.4393734ae917e967cf2b5b4b3b44ed0d.', STARTKEY => '', ENDKEY => ''}] 2023-04-19 20:56:04,258 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-19 20:56:04,258 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1681937764258"}]},"ts":"9223372036854775807"} 2023-04-19 20:56:04,259 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-19 20:56:04,262 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=53, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-19 20:56:04,263 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=53, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 33 msec 2023-04-19 20:56:04,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1227): Checking to see if procedure is done pid=53 2023-04-19 20:56:04,333 INFO [Listener at localhost/34893] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 53 completed 2023-04-19 20:56:04,345 INFO [Listener at localhost/34893] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testScanAtomicity Thread=230 (was 228) - Thread LEAK? -, OpenFileDescriptor=612 (was 561) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=415 (was 394) - SystemLoadAverage LEAK? -, ProcessCount=172 (was 172), AvailableMemoryMB=5649 (was 6274) 2023-04-19 20:56:04,357 INFO [Listener at localhost/34893] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobGetAtomicity Thread=230, OpenFileDescriptor=612, MaxFileDescriptor=60000, SystemLoadAverage=415, ProcessCount=172, AvailableMemoryMB=5649 2023-04-19 20:56:04,359 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-19 20:56:04,359 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-19 20:56:04,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure2.ProcedureExecutor(1029): Stored pid=54, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-19 20:56:04,361 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-19 20:56:04,361 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 54 2023-04-19 20:56:04,362 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-19 20:56:04,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-19 20:56:04,363 DEBUG [HFileArchiver-12] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/.tmp/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267 2023-04-19 20:56:04,363 DEBUG [HFileArchiver-12] backup.HFileArchiver(153): Directory hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/.tmp/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267 empty. 2023-04-19 20:56:04,364 DEBUG [HFileArchiver-12] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/.tmp/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267 2023-04-19 20:56:04,364 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-19 20:56:04,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-19 20:56:04,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-19 20:56:04,771 DEBUG [PEWorker-5] util.FSTableDescriptors(570): Wrote into hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-19 20:56:04,772 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 6873ba93789e63dd9f14391208147267, NAME => 'TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/.tmp 2023-04-19 20:56:04,780 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-19 20:56:04,780 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 6873ba93789e63dd9f14391208147267, disabling compactions & flushes 2023-04-19 20:56:04,780 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:04,780 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:04,780 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. after waiting 0 ms 2023-04-19 20:56:04,780 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:04,780 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:04,780 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 6873ba93789e63dd9f14391208147267: 2023-04-19 20:56:04,782 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-19 20:56:04,783 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681937764783"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681937764783"}]},"ts":"1681937764783"} 2023-04-19 20:56:04,784 INFO [PEWorker-5] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-19 20:56:04,784 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-19 20:56:04,785 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681937764784"}]},"ts":"1681937764784"} 2023-04-19 20:56:04,785 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-19 20:56:04,788 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=6873ba93789e63dd9f14391208147267, ASSIGN}] 2023-04-19 20:56:04,790 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=6873ba93789e63dd9f14391208147267, ASSIGN 2023-04-19 20:56:04,790 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=6873ba93789e63dd9f14391208147267, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,35247,1681937616971; forceNewPlan=false, retain=false 2023-04-19 20:56:04,942 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=6873ba93789e63dd9f14391208147267, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:04,942 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681937764942"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681937764942"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681937764942"}]},"ts":"1681937764942"} 2023-04-19 20:56:04,943 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=56, ppid=55, state=RUNNABLE; OpenRegionProcedure 6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971}] 2023-04-19 20:56:04,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-19 20:56:05,099 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:05,099 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 6873ba93789e63dd9f14391208147267, NAME => 'TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267.', STARTKEY => '', ENDKEY => ''} 2023-04-19 20:56:05,099 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 6873ba93789e63dd9f14391208147267 2023-04-19 20:56:05,100 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-19 20:56:05,100 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 6873ba93789e63dd9f14391208147267 2023-04-19 20:56:05,100 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 6873ba93789e63dd9f14391208147267 2023-04-19 20:56:05,101 INFO [StoreOpener-6873ba93789e63dd9f14391208147267-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 6873ba93789e63dd9f14391208147267 2023-04-19 20:56:05,102 DEBUG [StoreOpener-6873ba93789e63dd9f14391208147267-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A 2023-04-19 20:56:05,102 DEBUG [StoreOpener-6873ba93789e63dd9f14391208147267-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A 2023-04-19 20:56:05,103 INFO [StoreOpener-6873ba93789e63dd9f14391208147267-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-19 20:56:05,103 INFO [StoreOpener-6873ba93789e63dd9f14391208147267-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 6873ba93789e63dd9f14391208147267 columnFamilyName A 2023-04-19 20:56:05,103 INFO [StoreOpener-6873ba93789e63dd9f14391208147267-1] regionserver.HStore(310): Store=6873ba93789e63dd9f14391208147267/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-19 20:56:05,103 INFO [StoreOpener-6873ba93789e63dd9f14391208147267-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 6873ba93789e63dd9f14391208147267 2023-04-19 20:56:05,104 DEBUG [StoreOpener-6873ba93789e63dd9f14391208147267-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B 2023-04-19 20:56:05,104 DEBUG [StoreOpener-6873ba93789e63dd9f14391208147267-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B 2023-04-19 20:56:05,104 INFO [StoreOpener-6873ba93789e63dd9f14391208147267-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-19 20:56:05,105 INFO [StoreOpener-6873ba93789e63dd9f14391208147267-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 6873ba93789e63dd9f14391208147267 columnFamilyName B 2023-04-19 20:56:05,105 INFO [StoreOpener-6873ba93789e63dd9f14391208147267-1] regionserver.HStore(310): Store=6873ba93789e63dd9f14391208147267/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-19 20:56:05,105 INFO [StoreOpener-6873ba93789e63dd9f14391208147267-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 6873ba93789e63dd9f14391208147267 2023-04-19 20:56:05,106 DEBUG [StoreOpener-6873ba93789e63dd9f14391208147267-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C 2023-04-19 20:56:05,106 DEBUG [StoreOpener-6873ba93789e63dd9f14391208147267-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C 2023-04-19 20:56:05,106 INFO [StoreOpener-6873ba93789e63dd9f14391208147267-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-19 20:56:05,106 INFO [StoreOpener-6873ba93789e63dd9f14391208147267-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 6873ba93789e63dd9f14391208147267 columnFamilyName C 2023-04-19 20:56:05,107 INFO [StoreOpener-6873ba93789e63dd9f14391208147267-1] regionserver.HStore(310): Store=6873ba93789e63dd9f14391208147267/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-19 20:56:05,107 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:05,107 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267 2023-04-19 20:56:05,107 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267 2023-04-19 20:56:05,109 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-19 20:56:05,110 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 6873ba93789e63dd9f14391208147267 2023-04-19 20:56:05,111 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-19 20:56:05,111 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 6873ba93789e63dd9f14391208147267; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71587097, jitterRate=0.06673087179660797}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-19 20:56:05,111 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 6873ba93789e63dd9f14391208147267: 2023-04-19 20:56:05,112 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267., pid=56, masterSystemTime=1681937765094 2023-04-19 20:56:05,113 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:05,113 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:05,113 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=6873ba93789e63dd9f14391208147267, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:05,114 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681937765113"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681937765113"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681937765113"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681937765113"}]},"ts":"1681937765113"} 2023-04-19 20:56:05,115 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=56, resume processing ppid=55 2023-04-19 20:56:05,115 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=56, ppid=55, state=SUCCESS; OpenRegionProcedure 6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 in 171 msec 2023-04-19 20:56:05,116 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=55, resume processing ppid=54 2023-04-19 20:56:05,116 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=55, ppid=54, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=6873ba93789e63dd9f14391208147267, ASSIGN in 327 msec 2023-04-19 20:56:05,117 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-19 20:56:05,117 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681937765117"}]},"ts":"1681937765117"} 2023-04-19 20:56:05,118 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-19 20:56:05,120 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-19 20:56:05,121 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=54, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 761 msec 2023-04-19 20:56:05,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-19 20:56:05,466 INFO [Listener at localhost/34893] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 54 completed 2023-04-19 20:56:05,467 DEBUG [Listener at localhost/34893] zookeeper.ReadOnlyZKClient(139): Connect 0x0c0766f0 to 127.0.0.1:58736 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-19 20:56:05,473 DEBUG [Listener at localhost/34893] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4b13e8aa, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-19 20:56:05,474 DEBUG [Listener at localhost/34893] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-19 20:56:05,476 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:57162, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-19 20:56:05,477 DEBUG [Listener at localhost/34893] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-19 20:56:05,479 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:58674, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-19 20:56:05,480 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-19 20:56:05,481 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.HMaster$12(2454): Client=jenkins//172.31.14.131 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-19 20:56:05,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure2.ProcedureExecutor(1029): Stored pid=57, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-04-19 20:56:05,890 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-19 20:56:05,891 DEBUG [PEWorker-3] util.FSTableDescriptors(519): Deleted hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-19 20:56:05,891 INFO [PEWorker-3] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-19 20:56:05,893 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=58, ppid=57, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-04-19 20:56:05,895 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=6873ba93789e63dd9f14391208147267, REOPEN/MOVE}] 2023-04-19 20:56:05,896 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=6873ba93789e63dd9f14391208147267, REOPEN/MOVE 2023-04-19 20:56:05,897 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=6873ba93789e63dd9f14391208147267, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:05,897 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681937765897"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681937765897"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681937765897"}]},"ts":"1681937765897"} 2023-04-19 20:56:05,898 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=60, ppid=59, state=RUNNABLE; CloseRegionProcedure 6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971}] 2023-04-19 20:56:06,049 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 6873ba93789e63dd9f14391208147267 2023-04-19 20:56:06,050 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 6873ba93789e63dd9f14391208147267, disabling compactions & flushes 2023-04-19 20:56:06,050 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:06,050 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:06,050 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. after waiting 0 ms 2023-04-19 20:56:06,050 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:06,053 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-04-19 20:56:06,054 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:06,054 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 6873ba93789e63dd9f14391208147267: 2023-04-19 20:56:06,054 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(3509): Not adding moved region record: 6873ba93789e63dd9f14391208147267 to self. 2023-04-19 20:56:06,055 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 6873ba93789e63dd9f14391208147267 2023-04-19 20:56:06,056 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=6873ba93789e63dd9f14391208147267, regionState=CLOSED 2023-04-19 20:56:06,056 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681937766056"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681937766056"}]},"ts":"1681937766056"} 2023-04-19 20:56:06,058 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=60, resume processing ppid=59 2023-04-19 20:56:06,058 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=60, ppid=59, state=SUCCESS; CloseRegionProcedure 6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 in 159 msec 2023-04-19 20:56:06,058 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=6873ba93789e63dd9f14391208147267, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase4.apache.org,35247,1681937616971; forceNewPlan=false, retain=true 2023-04-19 20:56:06,209 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=6873ba93789e63dd9f14391208147267, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:06,209 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681937766209"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681937766209"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681937766209"}]},"ts":"1681937766209"} 2023-04-19 20:56:06,210 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=61, ppid=59, state=RUNNABLE; OpenRegionProcedure 6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971}] 2023-04-19 20:56:06,364 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:06,364 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 6873ba93789e63dd9f14391208147267, NAME => 'TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267.', STARTKEY => '', ENDKEY => ''} 2023-04-19 20:56:06,365 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 6873ba93789e63dd9f14391208147267 2023-04-19 20:56:06,365 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-19 20:56:06,365 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 6873ba93789e63dd9f14391208147267 2023-04-19 20:56:06,365 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 6873ba93789e63dd9f14391208147267 2023-04-19 20:56:06,366 INFO [StoreOpener-6873ba93789e63dd9f14391208147267-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 6873ba93789e63dd9f14391208147267 2023-04-19 20:56:06,367 DEBUG [StoreOpener-6873ba93789e63dd9f14391208147267-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A 2023-04-19 20:56:06,367 DEBUG [StoreOpener-6873ba93789e63dd9f14391208147267-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A 2023-04-19 20:56:06,367 INFO [StoreOpener-6873ba93789e63dd9f14391208147267-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-19 20:56:06,367 INFO [StoreOpener-6873ba93789e63dd9f14391208147267-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 6873ba93789e63dd9f14391208147267 columnFamilyName A 2023-04-19 20:56:06,368 INFO [StoreOpener-6873ba93789e63dd9f14391208147267-1] regionserver.HStore(310): Store=6873ba93789e63dd9f14391208147267/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-19 20:56:06,368 INFO [StoreOpener-6873ba93789e63dd9f14391208147267-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 6873ba93789e63dd9f14391208147267 2023-04-19 20:56:06,369 DEBUG [StoreOpener-6873ba93789e63dd9f14391208147267-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B 2023-04-19 20:56:06,369 DEBUG [StoreOpener-6873ba93789e63dd9f14391208147267-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B 2023-04-19 20:56:06,369 INFO [StoreOpener-6873ba93789e63dd9f14391208147267-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-19 20:56:06,369 INFO [StoreOpener-6873ba93789e63dd9f14391208147267-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 6873ba93789e63dd9f14391208147267 columnFamilyName B 2023-04-19 20:56:06,370 INFO [StoreOpener-6873ba93789e63dd9f14391208147267-1] regionserver.HStore(310): Store=6873ba93789e63dd9f14391208147267/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-19 20:56:06,370 INFO [StoreOpener-6873ba93789e63dd9f14391208147267-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 6873ba93789e63dd9f14391208147267 2023-04-19 20:56:06,370 DEBUG [StoreOpener-6873ba93789e63dd9f14391208147267-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C 2023-04-19 20:56:06,370 DEBUG [StoreOpener-6873ba93789e63dd9f14391208147267-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C 2023-04-19 20:56:06,371 INFO [StoreOpener-6873ba93789e63dd9f14391208147267-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-19 20:56:06,371 INFO [StoreOpener-6873ba93789e63dd9f14391208147267-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 6873ba93789e63dd9f14391208147267 columnFamilyName C 2023-04-19 20:56:06,371 INFO [StoreOpener-6873ba93789e63dd9f14391208147267-1] regionserver.HStore(310): Store=6873ba93789e63dd9f14391208147267/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-19 20:56:06,371 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:06,372 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267 2023-04-19 20:56:06,373 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267 2023-04-19 20:56:06,374 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-19 20:56:06,375 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 6873ba93789e63dd9f14391208147267 2023-04-19 20:56:06,376 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 6873ba93789e63dd9f14391208147267; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=65813425, jitterRate=-0.019303545355796814}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-19 20:56:06,376 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 6873ba93789e63dd9f14391208147267: 2023-04-19 20:56:06,377 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267., pid=61, masterSystemTime=1681937766361 2023-04-19 20:56:06,378 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:06,378 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:06,378 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=6873ba93789e63dd9f14391208147267, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:06,378 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681937766378"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681937766378"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681937766378"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681937766378"}]},"ts":"1681937766378"} 2023-04-19 20:56:06,380 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=61, resume processing ppid=59 2023-04-19 20:56:06,380 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=61, ppid=59, state=SUCCESS; OpenRegionProcedure 6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 in 169 msec 2023-04-19 20:56:06,381 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=59, resume processing ppid=58 2023-04-19 20:56:06,381 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=59, ppid=58, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=6873ba93789e63dd9f14391208147267, REOPEN/MOVE in 485 msec 2023-04-19 20:56:06,382 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=58, resume processing ppid=57 2023-04-19 20:56:06,383 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=58, ppid=57, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 488 msec 2023-04-19 20:56:06,384 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=57, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 902 msec 2023-04-19 20:56:06,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1227): Checking to see if procedure is done pid=57 2023-04-19 20:56:06,385 DEBUG [Listener at localhost/34893] zookeeper.ReadOnlyZKClient(139): Connect 0x6257294f to 127.0.0.1:58736 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-19 20:56:06,390 DEBUG [Listener at localhost/34893] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@55d0f313, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-19 20:56:06,391 DEBUG [Listener at localhost/34893] zookeeper.ReadOnlyZKClient(139): Connect 0x52104714 to 127.0.0.1:58736 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-19 20:56:06,397 DEBUG [Listener at localhost/34893] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@9d61148, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-19 20:56:06,397 DEBUG [Listener at localhost/34893] zookeeper.ReadOnlyZKClient(139): Connect 0x4a320817 to 127.0.0.1:58736 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-19 20:56:06,401 DEBUG [Listener at localhost/34893] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1a4acea2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-19 20:56:06,402 DEBUG [Listener at localhost/34893] zookeeper.ReadOnlyZKClient(139): Connect 0x19c1e3ee to 127.0.0.1:58736 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-19 20:56:06,406 DEBUG [Listener at localhost/34893] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@43693271, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-19 20:56:06,406 DEBUG [Listener at localhost/34893] zookeeper.ReadOnlyZKClient(139): Connect 0x526f2221 to 127.0.0.1:58736 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-19 20:56:06,410 DEBUG [Listener at localhost/34893] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@44277511, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-19 20:56:06,410 DEBUG [Listener at localhost/34893] zookeeper.ReadOnlyZKClient(139): Connect 0x3c1a68cd to 127.0.0.1:58736 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-19 20:56:06,414 DEBUG [Listener at localhost/34893] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@43ea1da2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-19 20:56:06,414 DEBUG [Listener at localhost/34893] zookeeper.ReadOnlyZKClient(139): Connect 0x5a709b21 to 127.0.0.1:58736 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-19 20:56:06,418 DEBUG [Listener at localhost/34893] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@35c3cca2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-19 20:56:06,419 DEBUG [Listener at localhost/34893] zookeeper.ReadOnlyZKClient(139): Connect 0x156e4ba6 to 127.0.0.1:58736 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-19 20:56:06,425 DEBUG [Listener at localhost/34893] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4b1973b9, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-19 20:56:06,425 DEBUG [Listener at localhost/34893] zookeeper.ReadOnlyZKClient(139): Connect 0x785e366b to 127.0.0.1:58736 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-19 20:56:06,429 DEBUG [Listener at localhost/34893] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1f0a71bd, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-19 20:56:06,429 DEBUG [Listener at localhost/34893] zookeeper.ReadOnlyZKClient(139): Connect 0x5fdfef3b to 127.0.0.1:58736 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-19 20:56:06,435 DEBUG [Listener at localhost/34893] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@56bb77c5, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-19 20:56:06,444 DEBUG [hconnection-0x4fb1e3cc-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-19 20:56:06,444 DEBUG [hconnection-0x65e45b43-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-19 20:56:06,444 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:56:06,449 DEBUG [hconnection-0x1c5fe6f-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-19 20:56:06,449 DEBUG [hconnection-0x33056dc4-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-19 20:56:06,450 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:57204, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-19 20:56:06,453 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:57176, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-19 20:56:06,456 DEBUG [hconnection-0x4249eaf6-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-19 20:56:06,457 DEBUG [hconnection-0x7c36a053-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-19 20:56:06,457 DEBUG [hconnection-0x1702f866-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-19 20:56:06,457 DEBUG [hconnection-0x660341a4-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-19 20:56:06,458 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:57164, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-19 20:56:06,457 DEBUG [hconnection-0x16e321fe-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-19 20:56:06,458 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:57192, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-19 20:56:06,458 DEBUG [hconnection-0xbc50a61-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-19 20:56:06,459 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:57210, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-19 20:56:06,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:56:06,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:56:06,461 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:56:06,461 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:56:06,461 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:56:06,461 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:56:06,461 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:57224, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-19 20:56:06,461 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:57236, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-19 20:56:06,461 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:06,461 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:06,466 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:57226, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-19 20:56:06,466 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:56:06,466 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:06,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:56:06,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:56:06,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:06,471 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:06,471 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:57228, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-19 20:56:06,471 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:56:06,471 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:57246, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-19 20:56:06,472 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:06,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:56:06,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:06,473 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:56:06,480 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:56:06,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(9158): Flush requested on 6873ba93789e63dd9f14391208147267 2023-04-19 20:56:06,481 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:56:06,481 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:56:06,481 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 6873ba93789e63dd9f14391208147267 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-19 20:56:06,481 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:56:06,481 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6873ba93789e63dd9f14391208147267, store=A 2023-04-19 20:56:06,481 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:56:06,481 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6873ba93789e63dd9f14391208147267, store=B 2023-04-19 20:56:06,481 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:56:06,481 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6873ba93789e63dd9f14391208147267, store=C 2023-04-19 20:56:06,481 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:56:06,485 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:56:06,485 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:56:06,485 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:06,485 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. started... 2023-04-19 20:56:06,485 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. as already flushing 2023-04-19 20:56:06,485 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:06,485 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:56:06,485 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:56:06,485 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:06,486 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:56:06,486 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:56:06,488 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:06,489 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:06,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:06,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:06,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:06,489 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:06,489 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:56:06,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:06,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:06,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:06,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:06,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:06,490 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:56:06,490 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@73135982[Count = 0] remaining members to acquire global barrier 2023-04-19 20:56:06,490 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:56:06,491 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:06,494 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:06,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:06,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:06,495 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:56:06,495 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:56:06,495 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:56:06,495 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:06,496 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:56:06,502 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:56:06,502 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:56:06,502 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:06,502 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:56:06,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:06,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:06,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:06,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:06,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:06,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:06,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:06,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:06,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:06,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:06,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:56:06,504 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:56:06,504 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:56:06,504 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:56:06,504 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:56:06,504 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:56:06,504 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:56:06,506 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:06,506 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:06,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:06,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:06,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:06,507 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:56:06,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:06,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:06,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:06,507 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:06,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:06,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:56:06,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:56:06,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:06,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:06,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:06,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:06,509 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:06,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:06,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:06,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:06,511 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:06,511 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:06,511 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:06,511 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:56:06,511 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:56:06,512 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:56:06,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:56:06,511 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:56:06,511 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:06,511 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:56:06,512 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:06,512 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:06,512 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:06,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:56:06,512 DEBUG [Thread-4043] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:56:06,512 DEBUG [Thread-4043] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:56:06,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:56:06,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:56:06,528 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:06,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57236 deadline: 1681937826526, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:06,529 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:06,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57210 deadline: 1681937826526, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:06,529 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:06,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57204 deadline: 1681937826526, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:06,530 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:06,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57226 deadline: 1681937826527, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:06,530 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:06,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57246 deadline: 1681937826528, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:06,612 DEBUG [Thread-4043] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:56:06,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:56:06,614 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:56:06,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:56:06,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:56:06,616 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:56:06,616 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:56:06,616 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:56:06,616 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:56:06,616 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:06,616 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:06,617 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:56:06,617 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:06,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:56:06,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:56:06,618 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:06,618 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:56:06,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:06,618 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:06,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:56:06,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:06,619 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:56:06,619 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:56:06,619 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:56:06,619 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:56:06,619 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:56:06,619 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:56:06,619 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:56:06,619 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:56:06,619 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:06,620 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. started... 2023-04-19 20:56:06,620 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. as already flushing 2023-04-19 20:56:06,620 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:06,620 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:56:06,620 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:56:06,620 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:06,620 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:56:06,620 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:56:06,621 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:06,621 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:06,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:06,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:06,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:06,622 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:06,622 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:56:06,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:06,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:06,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:06,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:06,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:06,623 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:56:06,623 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@deaf82a[Count = 0] remaining members to acquire global barrier 2023-04-19 20:56:06,623 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:56:06,623 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:06,626 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:06,626 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:06,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:06,626 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:56:06,626 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:06,626 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:56:06,626 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:56:06,626 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:56:06,628 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:06,628 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:56:06,628 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:56:06,628 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:56:06,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:06,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:06,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:06,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:06,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:06,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:06,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:06,630 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:06,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57236 deadline: 1681937826629, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:06,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:06,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:06,631 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:06,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57210 deadline: 1681937826630, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:06,631 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:06,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57204 deadline: 1681937826630, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:06,632 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:06,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57226 deadline: 1681937826631, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:06,633 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:06,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57246 deadline: 1681937826632, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:06,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:06,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:56:06,635 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:56:06,635 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:56:06,635 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:56:06,635 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:56:06,635 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:56:06,635 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:56:06,636 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:06,636 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:06,636 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:56:06,636 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:06,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:06,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:06,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:06,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:06,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:56:06,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:56:06,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:06,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:06,638 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:06,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:06,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:06,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:06,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:06,639 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:06,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:06,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:06,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:06,642 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:56:06,642 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:56:06,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:56:06,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:56:06,642 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:56:06,642 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:06,642 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:06,642 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:06,642 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:06,642 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:06,642 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:06,642 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:06,642 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:56:06,642 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:56:06,643 DEBUG [Thread-4043] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:56:06,643 DEBUG [Thread-4043] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:56:06,642 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:56:06,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:56:06,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:56:06,743 DEBUG [Thread-4043] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:56:06,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:56:06,745 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:56:06,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:56:06,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:56:06,746 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:56:06,746 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:56:06,746 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:56:06,746 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:56:06,748 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:06,748 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:06,750 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:06,750 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:56:06,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:56:06,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:56:06,751 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:06,751 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:56:06,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:06,751 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:06,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:56:06,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:06,752 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:56:06,754 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:56:06,754 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:56:06,754 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:56:06,754 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:56:06,754 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:56:06,755 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:56:06,755 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:56:06,755 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:06,755 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. started... 2023-04-19 20:56:06,755 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. as already flushing 2023-04-19 20:56:06,755 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:06,755 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:56:06,755 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:56:06,755 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:06,755 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:56:06,755 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:56:06,757 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:06,757 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:06,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:06,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:06,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:06,757 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:06,757 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:56:06,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:06,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:06,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:06,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:06,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:06,759 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:56:06,759 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@480e783d[Count = 0] remaining members to acquire global barrier 2023-04-19 20:56:06,759 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:56:06,759 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:06,760 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:06,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:06,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:06,760 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:56:06,760 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:56:06,760 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:56:06,760 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:06,760 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:56:06,762 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:06,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:06,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:06,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:06,762 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:56:06,762 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:56:06,762 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:56:06,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:06,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:06,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:06,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:06,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:06,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:06,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:06,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:56:06,764 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:56:06,764 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:56:06,764 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:56:06,764 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:56:06,764 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:56:06,764 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:56:06,766 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:06,766 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:06,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:06,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:06,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:06,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:06,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:06,766 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:56:06,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:56:06,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:56:06,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:06,767 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:06,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:06,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:06,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:06,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:06,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:06,768 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:06,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:06,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:06,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:06,770 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:06,770 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:56:06,770 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:06,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:56:06,770 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:56:06,770 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:56:06,770 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:56:06,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:56:06,770 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:56:06,771 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:56:06,770 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:06,771 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:06,771 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:06,771 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:06,771 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:06,771 DEBUG [Thread-4043] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:56:06,771 DEBUG [Thread-4043] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:56:06,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:56:06,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:56:06,833 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:06,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57236 deadline: 1681937826831, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:06,833 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:06,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57204 deadline: 1681937826832, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:06,835 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:06,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57210 deadline: 1681937826833, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:06,836 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:06,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57226 deadline: 1681937826834, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:06,836 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:06,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57246 deadline: 1681937826834, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:06,871 DEBUG [Thread-4043] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:56:06,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:56:06,873 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:56:06,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:56:06,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:56:06,875 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:56:06,875 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:56:06,875 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:56:06,875 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:56:06,875 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:06,875 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:06,876 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:56:06,876 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:56:06,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:56:06,876 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:06,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:06,877 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:06,877 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:56:06,877 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:06,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:56:06,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:06,878 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:56:06,878 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:56:06,878 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:56:06,878 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:56:06,878 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:56:06,878 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:56:06,878 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:56:06,878 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:56:06,878 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:06,878 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. started... 2023-04-19 20:56:06,878 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. as already flushing 2023-04-19 20:56:06,879 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:06,879 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:56:06,879 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:56:06,879 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:06,879 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:56:06,879 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:56:06,881 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:06,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:06,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:06,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:06,881 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:06,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:06,881 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:06,881 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:56:06,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:06,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:06,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:06,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:06,883 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:56:06,883 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6de832d8[Count = 0] remaining members to acquire global barrier 2023-04-19 20:56:06,883 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:56:06,883 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:06,886 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:06,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:06,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:06,886 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:56:06,886 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:56:06,886 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:06,886 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:56:06,886 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:56:06,887 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:06,887 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:56:06,887 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:56:06,887 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:56:06,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:06,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:06,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:06,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:06,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:06,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:06,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:06,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:06,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:06,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:06,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:56:06,891 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:56:06,891 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:56:06,891 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:56:06,891 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:56:06,891 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:56:06,891 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:56:06,895 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:06,895 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:56:06,896 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:06,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:06,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:06,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:06,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:06,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:06,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:06,896 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:06,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:56:06,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:06,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:56:06,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:06,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:06,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:06,898 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:06,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:06,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:06,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:06,901 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230419ecc0ceb8d0b743d0b126137582502c6a to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419ecc0ceb8d0b743d0b126137582502c6a 2023-04-19 20:56:06,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:06,904 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:56:06,904 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:56:06,904 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:56:06,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:56:06,904 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:06,904 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:06,904 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:06,905 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:06,905 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:06,905 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:56:06,905 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:56:06,905 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:06,905 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:06,905 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:56:06,905 DEBUG [Thread-4043] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:56:06,905 DEBUG [Thread-4043] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:56:06,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:56:06,905 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:56:06,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:56:06,908 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=16, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/A/ab63796c353f4994a363f7bc53116b16 2023-04-19 20:56:06,917 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=16 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/B/623b3d58bf8d4446897325a8d91870be 2023-04-19 20:56:06,929 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=16 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/C/e75d282e6e154426a1c0a12b4be75bd7 2023-04-19 20:56:06,932 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/A/ab63796c353f4994a363f7bc53116b16 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/ab63796c353f4994a363f7bc53116b16 2023-04-19 20:56:06,935 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/ab63796c353f4994a363f7bc53116b16, entries=150, sequenceid=16, filesize=25.1 K 2023-04-19 20:56:06,936 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/B/623b3d58bf8d4446897325a8d91870be as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/623b3d58bf8d4446897325a8d91870be 2023-04-19 20:56:06,939 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/623b3d58bf8d4446897325a8d91870be, entries=150, sequenceid=16, filesize=11.6 K 2023-04-19 20:56:06,940 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/C/e75d282e6e154426a1c0a12b4be75bd7 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/e75d282e6e154426a1c0a12b4be75bd7 2023-04-19 20:56:06,943 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/e75d282e6e154426a1c0a12b4be75bd7, entries=150, sequenceid=16, filesize=11.6 K 2023-04-19 20:56:06,944 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 6873ba93789e63dd9f14391208147267 in 463ms, sequenceid=16, compaction requested=false 2023-04-19 20:56:06,944 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 6873ba93789e63dd9f14391208147267: 2023-04-19 20:56:07,005 DEBUG [Thread-4043] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:56:07,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:56:07,007 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:56:07,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:56:07,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:56:07,009 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:56:07,009 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:56:07,009 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:56:07,009 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:56:07,009 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:07,009 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:07,010 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:56:07,010 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:07,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:56:07,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:56:07,011 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:07,011 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:56:07,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:07,011 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:07,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:56:07,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:07,012 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:56:07,012 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:56:07,012 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:56:07,012 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:56:07,012 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:56:07,012 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:56:07,013 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:56:07,013 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:56:07,013 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:07,013 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. started... 2023-04-19 20:56:07,013 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6873ba93789e63dd9f14391208147267 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-19 20:56:07,013 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6873ba93789e63dd9f14391208147267, store=A 2023-04-19 20:56:07,013 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:56:07,013 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6873ba93789e63dd9f14391208147267, store=B 2023-04-19 20:56:07,014 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:56:07,014 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6873ba93789e63dd9f14391208147267, store=C 2023-04-19 20:56:07,014 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:56:07,028 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230419635520dbce3844d4949094eaf511b5a5 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419635520dbce3844d4949094eaf511b5a5 2023-04-19 20:56:07,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(9158): Flush requested on 6873ba93789e63dd9f14391208147267 2023-04-19 20:56:07,137 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. as already flushing 2023-04-19 20:56:07,178 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:07,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57226 deadline: 1681937827175, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:07,178 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:07,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57204 deadline: 1681937827175, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:07,179 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:07,179 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:07,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57246 deadline: 1681937827175, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:07,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57236 deadline: 1681937827176, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:07,179 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:07,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57210 deadline: 1681937827177, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:07,280 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:07,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57226 deadline: 1681937827279, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:07,281 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:07,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57204 deadline: 1681937827280, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:07,281 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:07,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57246 deadline: 1681937827280, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:07,281 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:07,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57236 deadline: 1681937827280, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:07,281 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:07,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57210 deadline: 1681937827280, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:07,438 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=36, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/A/862c0865cb78432f8b2f2bf0bd08d25f 2023-04-19 20:56:07,446 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=36 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/B/74763615523f4ee895b7c97ef70298f5 2023-04-19 20:56:07,454 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=36 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/C/a79628ffa4934ee4afb6fde94c029cd0 2023-04-19 20:56:07,457 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/A/862c0865cb78432f8b2f2bf0bd08d25f as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/862c0865cb78432f8b2f2bf0bd08d25f 2023-04-19 20:56:07,461 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/862c0865cb78432f8b2f2bf0bd08d25f, entries=150, sequenceid=36, filesize=25.1 K 2023-04-19 20:56:07,461 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/B/74763615523f4ee895b7c97ef70298f5 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/74763615523f4ee895b7c97ef70298f5 2023-04-19 20:56:07,465 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/74763615523f4ee895b7c97ef70298f5, entries=150, sequenceid=36, filesize=11.6 K 2023-04-19 20:56:07,465 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/C/a79628ffa4934ee4afb6fde94c029cd0 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/a79628ffa4934ee4afb6fde94c029cd0 2023-04-19 20:56:07,469 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/a79628ffa4934ee4afb6fde94c029cd0, entries=150, sequenceid=36, filesize=11.6 K 2023-04-19 20:56:07,469 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 6873ba93789e63dd9f14391208147267 in 456ms, sequenceid=36, compaction requested=false 2023-04-19 20:56:07,469 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6873ba93789e63dd9f14391208147267: 2023-04-19 20:56:07,469 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:07,469 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:56:07,470 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:56:07,470 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:07,470 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:56:07,470 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:56:07,471 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:07,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:07,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:07,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:07,471 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:07,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:07,472 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:07,472 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:56:07,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:07,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:07,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:07,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:07,473 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:56:07,473 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4deb9ae[Count = 0] remaining members to acquire global barrier 2023-04-19 20:56:07,473 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:56:07,473 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:07,474 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:07,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:07,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:07,475 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:56:07,475 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:07,475 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:56:07,475 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:56:07,475 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:56:07,477 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:56:07,477 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:56:07,477 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:07,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:07,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:07,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:07,477 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:56:07,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:07,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:07,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:07,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:07,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:07,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:07,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:07,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:56:07,480 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:56:07,480 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:56:07,481 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:56:07,481 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:56:07,481 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:56:07,481 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:56:07,482 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:07,482 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:56:07,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:07,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:07,483 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:07,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:07,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:07,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:07,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:56:07,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:56:07,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(9158): Flush requested on 6873ba93789e63dd9f14391208147267 2023-04-19 20:56:07,484 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 6873ba93789e63dd9f14391208147267 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-19 20:56:07,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:07,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:07,485 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:07,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:07,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:07,486 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6873ba93789e63dd9f14391208147267, store=A 2023-04-19 20:56:07,486 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:56:07,486 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6873ba93789e63dd9f14391208147267, store=B 2023-04-19 20:56:07,486 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:56:07,486 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6873ba93789e63dd9f14391208147267, store=C 2023-04-19 20:56:07,486 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:56:07,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:07,487 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:07,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:07,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:07,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:07,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:07,490 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:07,490 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:07,490 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:07,490 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:56:07,490 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:07,490 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:56:07,490 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:56:07,490 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:56:07,490 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:07,490 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:56:07,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:56:07,490 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:07,490 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:07,490 DEBUG [Thread-4043] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:56:07,490 DEBUG [Thread-4043] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:56:07,490 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:56:07,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:56:07,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:56:07,491 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:56:07,501 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:07,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57204 deadline: 1681937827497, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:07,502 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:07,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57246 deadline: 1681937827498, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:07,503 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:07,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57236 deadline: 1681937827499, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:07,503 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:07,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57226 deadline: 1681937827499, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:07,503 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:07,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57210 deadline: 1681937827500, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:07,591 DEBUG [Thread-4043] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:56:07,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:56:07,593 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:56:07,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:56:07,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:56:07,594 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:56:07,594 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:56:07,594 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:56:07,594 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:56:07,594 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:07,595 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:07,596 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:56:07,596 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:07,596 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:56:07,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:56:07,596 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:07,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:07,596 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:56:07,597 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:07,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:56:07,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:07,597 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:56:07,597 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:56:07,597 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:56:07,597 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:56:07,597 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:56:07,597 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:56:07,598 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:56:07,598 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:56:07,598 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:07,598 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. started... 2023-04-19 20:56:07,598 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. as already flushing 2023-04-19 20:56:07,598 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:07,598 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:56:07,598 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:56:07,598 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:07,598 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:56:07,598 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:56:07,601 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:07,601 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:07,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:07,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:07,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:07,601 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:07,601 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:56:07,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:07,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:07,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:07,602 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:07,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57204 deadline: 1681937827602, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:07,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:07,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:07,603 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:56:07,603 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5e2f3ab7[Count = 0] remaining members to acquire global barrier 2023-04-19 20:56:07,603 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:56:07,603 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:07,605 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:07,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:07,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:07,605 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:56:07,605 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:56:07,605 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:56:07,605 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:07,605 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:56:07,605 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:07,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57246 deadline: 1681937827604, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:07,606 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:07,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57236 deadline: 1681937827604, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:07,606 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:07,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57226 deadline: 1681937827604, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:07,606 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:07,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57210 deadline: 1681937827604, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:07,607 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:56:07,607 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:07,607 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:56:07,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:07,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:07,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:07,607 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:56:07,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:07,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:07,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:07,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:07,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:07,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:07,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:07,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:56:07,610 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:56:07,610 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:56:07,610 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:56:07,610 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:56:07,610 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:56:07,610 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:56:07,611 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:07,611 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:07,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:07,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:07,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:07,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:07,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:07,611 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:56:07,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:56:07,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:56:07,612 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:07,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:07,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:07,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:07,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:07,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:07,613 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:07,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:07,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:07,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:07,617 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:07,617 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:56:07,617 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:07,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:56:07,617 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:07,617 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:56:07,617 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:56:07,617 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:56:07,617 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:56:07,617 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:07,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:56:07,617 DEBUG [Thread-4043] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:56:07,617 DEBUG [Thread-4043] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:56:07,617 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:56:07,617 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:07,617 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:07,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:56:07,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:56:07,617 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:07,717 DEBUG [Thread-4043] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:56:07,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:56:07,719 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:56:07,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:56:07,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:56:07,721 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:56:07,721 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:56:07,721 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:56:07,721 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:56:07,722 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:07,722 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:07,723 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:07,723 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:56:07,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:56:07,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:56:07,724 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:07,724 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:56:07,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:07,725 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:07,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:56:07,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:07,725 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:56:07,725 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:56:07,725 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:56:07,725 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:56:07,725 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:56:07,725 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:56:07,726 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:56:07,726 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:56:07,726 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:07,726 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. started... 2023-04-19 20:56:07,726 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. as already flushing 2023-04-19 20:56:07,726 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:07,726 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:56:07,726 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:56:07,726 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:07,726 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:56:07,726 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:56:07,728 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:07,728 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:07,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:07,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:07,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:07,728 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:07,728 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:56:07,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:07,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:07,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:07,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:07,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:07,730 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:56:07,730 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@655b7edf[Count = 0] remaining members to acquire global barrier 2023-04-19 20:56:07,730 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:56:07,730 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:07,731 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:07,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:07,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:07,731 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:07,731 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:56:07,731 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:56:07,731 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:56:07,731 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:56:07,733 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:07,733 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:56:07,734 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:56:07,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:07,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:07,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:07,734 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:56:07,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:07,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:07,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:07,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:07,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:07,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:07,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:07,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:56:07,736 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:56:07,736 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:56:07,736 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:56:07,736 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:56:07,736 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:56:07,737 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:56:07,738 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:07,738 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:56:07,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:07,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:07,738 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:07,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:56:07,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:56:07,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:07,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:07,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:07,739 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:07,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:07,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:07,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:07,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:07,741 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:07,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:07,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:07,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:07,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:07,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:07,745 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:07,745 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:07,745 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:56:07,745 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:56:07,745 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:07,746 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:56:07,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:56:07,746 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:56:07,745 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:56:07,746 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:56:07,746 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:56:07,746 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:07,746 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:07,746 DEBUG [Thread-4043] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:56:07,746 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:07,746 DEBUG [Thread-4043] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:56:07,746 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:07,746 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:56:07,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:56:07,805 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:07,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57204 deadline: 1681937827804, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:07,808 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:07,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57246 deadline: 1681937827808, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:07,809 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:07,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57226 deadline: 1681937827808, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:07,809 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:07,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57236 deadline: 1681937827808, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:07,809 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:07,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57210 deadline: 1681937827808, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:07,846 DEBUG [Thread-4043] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:56:07,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:56:07,848 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:56:07,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:56:07,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:56:07,850 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:56:07,850 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:56:07,850 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:56:07,850 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:56:07,850 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:07,850 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:07,851 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:07,851 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:56:07,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:56:07,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:56:07,852 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:07,852 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:56:07,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:07,852 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:07,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:56:07,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:07,853 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:56:07,853 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:56:07,853 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:56:07,853 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:56:07,853 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:56:07,853 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:56:07,853 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:56:07,853 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:56:07,853 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:07,853 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. started... 2023-04-19 20:56:07,853 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. as already flushing 2023-04-19 20:56:07,853 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:07,853 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:56:07,853 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:56:07,854 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:07,854 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:56:07,854 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:56:07,855 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:07,855 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:07,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:07,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:07,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:07,855 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:07,855 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:56:07,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:07,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:07,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:07,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:07,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:07,857 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:56:07,857 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4989fe5e[Count = 0] remaining members to acquire global barrier 2023-04-19 20:56:07,857 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:56:07,857 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:07,858 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:07,858 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:07,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:07,858 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:56:07,859 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:56:07,859 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:56:07,859 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:07,859 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:56:07,860 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:07,860 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:56:07,860 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:56:07,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:07,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:07,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:07,860 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:56:07,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:07,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:07,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:07,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:07,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:07,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:07,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:07,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:56:07,862 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:56:07,862 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:56:07,862 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:56:07,862 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:56:07,862 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:56:07,862 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:56:07,864 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:07,864 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:07,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:07,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:07,864 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:56:07,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:07,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:07,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:07,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:56:07,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:56:07,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:07,865 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:07,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:07,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:07,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:07,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:07,866 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:07,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:07,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:07,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:07,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:07,868 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:07,868 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:07,868 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:07,868 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:07,868 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:07,868 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:56:07,868 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:56:07,868 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:56:07,868 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:56:07,868 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:07,868 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:56:07,868 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:56:07,868 DEBUG [Thread-4043] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:56:07,868 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:56:07,869 DEBUG [Thread-4043] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:56:07,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:56:07,868 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:07,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:56:07,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:56:07,905 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304195c6c9e2130714c219c21147c92d97a32 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304195c6c9e2130714c219c21147c92d97a32 2023-04-19 20:56:07,910 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=52, memsize=29.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/A/f36f8c2b86634c97b8ffb1851c04b27a 2023-04-19 20:56:07,919 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=52 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/B/f268e27fcfa1457499b076a31a8d3e16 2023-04-19 20:56:07,940 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=52 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/C/31d9e9c62f0242e8bf31e58e758240f1 2023-04-19 20:56:07,944 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/A/f36f8c2b86634c97b8ffb1851c04b27a as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/f36f8c2b86634c97b8ffb1851c04b27a 2023-04-19 20:56:07,948 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/f36f8c2b86634c97b8ffb1851c04b27a, entries=150, sequenceid=52, filesize=25.1 K 2023-04-19 20:56:07,949 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/B/f268e27fcfa1457499b076a31a8d3e16 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/f268e27fcfa1457499b076a31a8d3e16 2023-04-19 20:56:07,952 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/f268e27fcfa1457499b076a31a8d3e16, entries=150, sequenceid=52, filesize=11.6 K 2023-04-19 20:56:07,953 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/C/31d9e9c62f0242e8bf31e58e758240f1 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/31d9e9c62f0242e8bf31e58e758240f1 2023-04-19 20:56:07,969 DEBUG [Thread-4043] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:56:07,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:56:07,971 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:56:07,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:56:07,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:56:07,972 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:56:07,972 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:56:07,972 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:56:07,972 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:56:07,972 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:07,973 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:07,973 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/31d9e9c62f0242e8bf31e58e758240f1, entries=150, sequenceid=52, filesize=11.6 K 2023-04-19 20:56:07,974 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 6873ba93789e63dd9f14391208147267 in 490ms, sequenceid=52, compaction requested=true 2023-04-19 20:56:07,974 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 6873ba93789e63dd9f14391208147267: 2023-04-19 20:56:07,974 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 20:56:07,974 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-19 20:56:07,974 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-19 20:56:07,974 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 20:56:07,974 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 20:56:07,975 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:56:07,975 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:07,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:56:07,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:56:07,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:07,975 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:07,975 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:56:07,976 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 77238 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-19 20:56:07,976 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-19 20:56:07,976 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1912): 6873ba93789e63dd9f14391208147267/A is initiating minor compaction (all files) 2023-04-19 20:56:07,976 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1912): 6873ba93789e63dd9f14391208147267/B is initiating minor compaction (all files) 2023-04-19 20:56:07,976 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 6873ba93789e63dd9f14391208147267/A in TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:07,976 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6873ba93789e63dd9f14391208147267/B in TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:07,976 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/ab63796c353f4994a363f7bc53116b16, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/862c0865cb78432f8b2f2bf0bd08d25f, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/f36f8c2b86634c97b8ffb1851c04b27a] into tmpdir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp, totalSize=75.4 K 2023-04-19 20:56:07,976 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/623b3d58bf8d4446897325a8d91870be, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/74763615523f4ee895b7c97ef70298f5, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/f268e27fcfa1457499b076a31a8d3e16] into tmpdir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp, totalSize=34.8 K 2023-04-19 20:56:07,976 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:07,976 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting ab63796c353f4994a363f7bc53116b16, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=16, earliestPutTs=1681937766475 2023-04-19 20:56:07,977 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 623b3d58bf8d4446897325a8d91870be, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=16, earliestPutTs=1681937766475 2023-04-19 20:56:07,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:56:07,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:07,977 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:56:07,977 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 862c0865cb78432f8b2f2bf0bd08d25f, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=36, earliestPutTs=1681937766525 2023-04-19 20:56:07,977 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 74763615523f4ee895b7c97ef70298f5, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=36, earliestPutTs=1681937766525 2023-04-19 20:56:07,977 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:56:07,977 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:56:07,977 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting f36f8c2b86634c97b8ffb1851c04b27a, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=52, earliestPutTs=1681937767175 2023-04-19 20:56:07,978 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting f268e27fcfa1457499b076a31a8d3e16, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=52, earliestPutTs=1681937767175 2023-04-19 20:56:07,977 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:56:07,978 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:56:07,978 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:56:07,978 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:56:07,978 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:56:07,978 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:07,978 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. started... 2023-04-19 20:56:07,978 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6873ba93789e63dd9f14391208147267 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-19 20:56:07,978 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6873ba93789e63dd9f14391208147267, store=A 2023-04-19 20:56:07,979 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:56:07,979 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6873ba93789e63dd9f14391208147267, store=B 2023-04-19 20:56:07,979 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:56:07,979 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6873ba93789e63dd9f14391208147267, store=C 2023-04-19 20:56:07,979 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:56:07,997 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] throttle.PressureAwareThroughputController(145): 6873ba93789e63dd9f14391208147267#A#compaction#552 average throughput is 1.52 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 20:56:08,000 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] throttle.PressureAwareThroughputController(145): 6873ba93789e63dd9f14391208147267#B#compaction#554 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 20:56:08,013 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041989499b73be374f819684e66b46a3595d to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041989499b73be374f819684e66b46a3595d 2023-04-19 20:56:08,028 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=68, memsize=29.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/A/8f95ea25964a4a5cb50534c22f1f2dfb 2023-04-19 20:56:08,046 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/A/eec6a4efa75342c8aaa239f120fc5773 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/eec6a4efa75342c8aaa239f120fc5773 2023-04-19 20:56:08,049 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 6873ba93789e63dd9f14391208147267/A of 6873ba93789e63dd9f14391208147267 into eec6a4efa75342c8aaa239f120fc5773(size=25.2 K), total size for store is 25.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 20:56:08,049 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6873ba93789e63dd9f14391208147267: 2023-04-19 20:56:08,049 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267., storeName=6873ba93789e63dd9f14391208147267/A, priority=13, startTime=1681937767974; duration=0sec 2023-04-19 20:56:08,050 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 20:56:08,050 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-19 20:56:08,051 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-19 20:56:08,051 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1912): 6873ba93789e63dd9f14391208147267/C is initiating minor compaction (all files) 2023-04-19 20:56:08,051 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 6873ba93789e63dd9f14391208147267/C in TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:08,051 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/e75d282e6e154426a1c0a12b4be75bd7, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/a79628ffa4934ee4afb6fde94c029cd0, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/31d9e9c62f0242e8bf31e58e758240f1] into tmpdir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp, totalSize=34.8 K 2023-04-19 20:56:08,052 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting e75d282e6e154426a1c0a12b4be75bd7, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=16, earliestPutTs=1681937766475 2023-04-19 20:56:08,052 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting a79628ffa4934ee4afb6fde94c029cd0, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=36, earliestPutTs=1681937766525 2023-04-19 20:56:08,052 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 31d9e9c62f0242e8bf31e58e758240f1, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=52, earliestPutTs=1681937767175 2023-04-19 20:56:08,059 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] throttle.PressureAwareThroughputController(145): 6873ba93789e63dd9f14391208147267#C#compaction#556 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 20:56:08,070 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/C/459f086d2f8a4a988a5447fbb85d6d74 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/459f086d2f8a4a988a5447fbb85d6d74 2023-04-19 20:56:08,074 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 6873ba93789e63dd9f14391208147267/C of 6873ba93789e63dd9f14391208147267 into 459f086d2f8a4a988a5447fbb85d6d74(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 20:56:08,074 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6873ba93789e63dd9f14391208147267: 2023-04-19 20:56:08,075 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267., storeName=6873ba93789e63dd9f14391208147267/C, priority=13, startTime=1681937767974; duration=0sec 2023-04-19 20:56:08,075 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 20:56:08,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(9158): Flush requested on 6873ba93789e63dd9f14391208147267 2023-04-19 20:56:08,108 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. as already flushing 2023-04-19 20:56:08,124 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:08,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57226 deadline: 1681937828120, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:08,124 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:08,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57204 deadline: 1681937828120, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:08,124 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:08,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57210 deadline: 1681937828121, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:08,125 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:08,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57246 deadline: 1681937828121, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:08,125 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:08,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57236 deadline: 1681937828122, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:08,227 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:08,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57226 deadline: 1681937828225, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:08,227 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:08,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57204 deadline: 1681937828225, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:08,228 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:08,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57210 deadline: 1681937828226, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:08,229 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:08,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57246 deadline: 1681937828226, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:08,229 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:08,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57236 deadline: 1681937828226, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:08,422 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/B/d3ae7695e34940de99097e297a7035ae as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/d3ae7695e34940de99097e297a7035ae 2023-04-19 20:56:08,426 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 6873ba93789e63dd9f14391208147267/B of 6873ba93789e63dd9f14391208147267 into d3ae7695e34940de99097e297a7035ae(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 20:56:08,426 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6873ba93789e63dd9f14391208147267: 2023-04-19 20:56:08,426 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267., storeName=6873ba93789e63dd9f14391208147267/B, priority=13, startTime=1681937767974; duration=0sec 2023-04-19 20:56:08,426 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 20:56:08,429 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:08,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57226 deadline: 1681937828428, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:08,431 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:08,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57204 deadline: 1681937828429, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:08,433 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:08,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57210 deadline: 1681937828430, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:08,433 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:08,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57246 deadline: 1681937828430, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:08,433 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:08,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57236 deadline: 1681937828431, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:08,442 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=68 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/B/d0eeacf0aa3f452b912d0c4cf3a29642 2023-04-19 20:56:08,453 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=68 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/C/ec555bd2579447849138ddb079eeef00 2023-04-19 20:56:08,458 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/A/8f95ea25964a4a5cb50534c22f1f2dfb as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/8f95ea25964a4a5cb50534c22f1f2dfb 2023-04-19 20:56:08,462 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/8f95ea25964a4a5cb50534c22f1f2dfb, entries=150, sequenceid=68, filesize=25.1 K 2023-04-19 20:56:08,462 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/B/d0eeacf0aa3f452b912d0c4cf3a29642 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/d0eeacf0aa3f452b912d0c4cf3a29642 2023-04-19 20:56:08,465 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/d0eeacf0aa3f452b912d0c4cf3a29642, entries=150, sequenceid=68, filesize=11.6 K 2023-04-19 20:56:08,466 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/C/ec555bd2579447849138ddb079eeef00 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/ec555bd2579447849138ddb079eeef00 2023-04-19 20:56:08,469 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/ec555bd2579447849138ddb079eeef00, entries=150, sequenceid=68, filesize=11.6 K 2023-04-19 20:56:08,470 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 6873ba93789e63dd9f14391208147267 in 492ms, sequenceid=68, compaction requested=false 2023-04-19 20:56:08,470 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6873ba93789e63dd9f14391208147267: 2023-04-19 20:56:08,470 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:08,470 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:56:08,470 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:56:08,470 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:08,470 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:56:08,470 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:56:08,472 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:08,472 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:08,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:08,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:08,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:08,472 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:08,472 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:56:08,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:08,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:08,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:08,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:08,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:08,474 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:56:08,474 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@386b3d08[Count = 0] remaining members to acquire global barrier 2023-04-19 20:56:08,474 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:56:08,474 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:08,476 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:08,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:08,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:08,476 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:08,476 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:56:08,476 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:56:08,476 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:56:08,476 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:56:08,478 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:56:08,478 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:56:08,478 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:08,478 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:56:08,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:08,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:08,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:08,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:08,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:08,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:08,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:08,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:08,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:08,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:08,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:56:08,481 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:56:08,481 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:56:08,481 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:56:08,481 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:56:08,481 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:56:08,481 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:56:08,483 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:08,483 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:08,483 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:56:08,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:08,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:08,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:08,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:08,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:08,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:08,484 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:08,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:56:08,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:56:08,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:08,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:08,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:08,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:08,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:08,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:08,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:08,486 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:08,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:08,488 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:56:08,488 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:56:08,488 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:08,489 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:56:08,489 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:56:08,489 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:08,488 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:56:08,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:56:08,489 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:08,489 DEBUG [Thread-4043] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:56:08,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:56:08,489 DEBUG [Thread-4043] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:56:08,489 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:08,489 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:08,489 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:08,489 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:08,489 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:56:08,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:56:08,589 DEBUG [Thread-4043] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:56:08,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:56:08,591 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:56:08,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:56:08,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:56:08,592 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:56:08,593 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:56:08,593 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:56:08,593 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:56:08,593 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:08,593 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:08,594 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:08,594 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:56:08,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:56:08,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:56:08,595 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:08,595 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:56:08,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:08,595 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:08,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:56:08,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:08,596 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:56:08,596 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:56:08,596 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:56:08,596 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:56:08,596 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:56:08,596 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:56:08,597 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:56:08,597 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:56:08,597 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:08,597 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. started... 2023-04-19 20:56:08,597 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6873ba93789e63dd9f14391208147267 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-19 20:56:08,597 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6873ba93789e63dd9f14391208147267, store=A 2023-04-19 20:56:08,597 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:56:08,597 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6873ba93789e63dd9f14391208147267, store=B 2023-04-19 20:56:08,597 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:56:08,597 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6873ba93789e63dd9f14391208147267, store=C 2023-04-19 20:56:08,597 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:56:08,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(9158): Flush requested on 6873ba93789e63dd9f14391208147267 2023-04-19 20:56:08,735 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. as already flushing 2023-04-19 20:56:08,771 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:08,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57226 deadline: 1681937828768, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:08,773 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:08,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57204 deadline: 1681937828769, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:08,773 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:08,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57210 deadline: 1681937828769, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:08,774 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:08,774 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:08,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57246 deadline: 1681937828769, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:08,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57236 deadline: 1681937828770, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:08,875 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:08,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57226 deadline: 1681937828872, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:08,876 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:08,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57204 deadline: 1681937828874, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:08,876 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:08,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57210 deadline: 1681937828874, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:08,877 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:08,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57236 deadline: 1681937828875, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:08,877 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:08,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57246 deadline: 1681937828875, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:09,021 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230419789f271c201d4003af84f5f78c11bf4d to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419789f271c201d4003af84f5f78c11bf4d 2023-04-19 20:56:09,078 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:09,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57226 deadline: 1681937829076, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:09,079 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:09,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57204 deadline: 1681937829077, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:09,079 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:09,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57210 deadline: 1681937829077, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:09,080 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:09,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57246 deadline: 1681937829078, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:09,082 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:09,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57236 deadline: 1681937829078, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:09,219 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-19 20:56:09,383 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:09,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57204 deadline: 1681937829380, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:09,383 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:09,383 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:09,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57226 deadline: 1681937829381, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:09,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57210 deadline: 1681937829381, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:09,383 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:09,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57246 deadline: 1681937829382, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:09,383 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:09,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57236 deadline: 1681937829382, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:09,427 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=87, memsize=29.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/A/f8f1c07250574ed481e0dca56a8de79c 2023-04-19 20:56:09,838 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=87 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/B/2d48a2781f9b434688b5d141f60c78a2 2023-04-19 20:56:09,884 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:09,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57210 deadline: 1681937829884, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:09,887 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:09,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57246 deadline: 1681937829884, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:09,888 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:09,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57236 deadline: 1681937829886, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:09,888 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:09,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57204 deadline: 1681937829887, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:09,888 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:09,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57226 deadline: 1681937829887, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:10,255 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=87 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/C/66ec2c0da0ab4ddfb833b817a8649e63 2023-04-19 20:56:10,259 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/A/f8f1c07250574ed481e0dca56a8de79c as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/f8f1c07250574ed481e0dca56a8de79c 2023-04-19 20:56:10,263 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/f8f1c07250574ed481e0dca56a8de79c, entries=150, sequenceid=87, filesize=25.1 K 2023-04-19 20:56:10,264 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/B/2d48a2781f9b434688b5d141f60c78a2 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/2d48a2781f9b434688b5d141f60c78a2 2023-04-19 20:56:10,267 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/2d48a2781f9b434688b5d141f60c78a2, entries=150, sequenceid=87, filesize=11.6 K 2023-04-19 20:56:10,268 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/C/66ec2c0da0ab4ddfb833b817a8649e63 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/66ec2c0da0ab4ddfb833b817a8649e63 2023-04-19 20:56:10,271 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/66ec2c0da0ab4ddfb833b817a8649e63, entries=150, sequenceid=87, filesize=11.6 K 2023-04-19 20:56:10,271 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=100.63 KB/103050 for 6873ba93789e63dd9f14391208147267 in 1674ms, sequenceid=87, compaction requested=true 2023-04-19 20:56:10,272 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6873ba93789e63dd9f14391208147267: 2023-04-19 20:56:10,272 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:10,272 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:56:10,272 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:56:10,272 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:10,272 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:56:10,272 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:56:10,273 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:10,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:10,273 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:10,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:10,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:10,274 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:10,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:10,274 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:56:10,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:10,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:10,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:10,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:10,275 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:56:10,275 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1f364230[Count = 0] remaining members to acquire global barrier 2023-04-19 20:56:10,275 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:56:10,275 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:10,276 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:10,276 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:10,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:10,276 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:56:10,277 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:56:10,277 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:56:10,277 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:10,277 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:56:10,288 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:56:10,288 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:56:10,288 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:56:10,289 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:10,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:10,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:10,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:10,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:10,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:10,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:10,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:10,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:10,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:10,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:10,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:56:10,292 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:56:10,292 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:56:10,292 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:56:10,292 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:56:10,292 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:56:10,292 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:56:10,295 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:10,295 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:10,295 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:56:10,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:10,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:10,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:10,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:10,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:10,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:56:10,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:56:10,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:10,296 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:10,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:10,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:10,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:10,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:10,297 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:10,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:10,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:10,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:10,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:10,303 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:56:10,303 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:10,303 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:56:10,303 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:56:10,303 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:56:10,303 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:10,303 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:10,303 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:10,303 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:10,303 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:10,303 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:10,303 DEBUG [Thread-4043] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:56:10,303 DEBUG [Thread-4043] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:56:10,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:56:10,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:56:10,303 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:56:10,303 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:56:10,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:56:10,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:56:10,403 DEBUG [Thread-4043] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:56:10,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:56:10,405 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:56:10,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:56:10,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:56:10,407 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:56:10,407 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:56:10,407 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:56:10,407 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:56:10,407 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:10,407 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:10,409 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:10,409 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:56:10,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:56:10,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:56:10,409 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:10,409 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:56:10,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:10,409 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:10,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:56:10,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:10,410 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:56:10,410 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:56:10,410 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:56:10,410 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:56:10,410 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:56:10,410 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:56:10,410 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:56:10,410 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:56:10,410 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:10,411 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. started... 2023-04-19 20:56:10,411 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6873ba93789e63dd9f14391208147267 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-19 20:56:10,411 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6873ba93789e63dd9f14391208147267, store=A 2023-04-19 20:56:10,411 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:56:10,412 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6873ba93789e63dd9f14391208147267, store=B 2023-04-19 20:56:10,412 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:56:10,412 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6873ba93789e63dd9f14391208147267, store=C 2023-04-19 20:56:10,412 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:56:10,429 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230419f500c071ce954f7abccf50c691ec2a57 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419f500c071ce954f7abccf50c691ec2a57 2023-04-19 20:56:10,434 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=105, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/A/3fa83711bf924f2f80a381c4fa673d33 2023-04-19 20:56:10,444 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=105 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/B/9e60c267bda7478ba55d83e91c3da20b 2023-04-19 20:56:10,856 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=105 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/C/5c6e274922f44ba7aa8a69aced8334cb 2023-04-19 20:56:10,860 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/A/3fa83711bf924f2f80a381c4fa673d33 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/3fa83711bf924f2f80a381c4fa673d33 2023-04-19 20:56:10,863 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/3fa83711bf924f2f80a381c4fa673d33, entries=150, sequenceid=105, filesize=25.1 K 2023-04-19 20:56:10,864 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/B/9e60c267bda7478ba55d83e91c3da20b as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/9e60c267bda7478ba55d83e91c3da20b 2023-04-19 20:56:10,868 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/9e60c267bda7478ba55d83e91c3da20b, entries=150, sequenceid=105, filesize=11.6 K 2023-04-19 20:56:10,868 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/C/5c6e274922f44ba7aa8a69aced8334cb as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/5c6e274922f44ba7aa8a69aced8334cb 2023-04-19 20:56:10,871 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/5c6e274922f44ba7aa8a69aced8334cb, entries=150, sequenceid=105, filesize=11.6 K 2023-04-19 20:56:10,872 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=0 B/0 for 6873ba93789e63dd9f14391208147267 in 461ms, sequenceid=105, compaction requested=true 2023-04-19 20:56:10,872 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6873ba93789e63dd9f14391208147267: 2023-04-19 20:56:10,872 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:10,872 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:56:10,872 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:56:10,872 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:10,872 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:56:10,872 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:56:10,874 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:10,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:10,874 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:10,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:10,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:10,875 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:10,875 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:56:10,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:10,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:10,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:10,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:10,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:10,876 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:56:10,876 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@21257a8f[Count = 0] remaining members to acquire global barrier 2023-04-19 20:56:10,876 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:56:10,876 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:10,878 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:10,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:10,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:10,879 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:56:10,879 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:10,879 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:56:10,879 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:56:10,879 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:56:10,880 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:56:10,880 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:56:10,880 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:10,880 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:56:10,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:10,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:10,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:10,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:10,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:10,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:10,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:10,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:10,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:10,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:10,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:56:10,883 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:56:10,883 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:56:10,883 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:56:10,883 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:56:10,883 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:56:10,883 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:56:10,885 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:10,885 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:10,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:10,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:10,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:10,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:10,885 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:56:10,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:10,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:56:10,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:10,885 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:10,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:56:10,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:10,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:10,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:10,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:10,887 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:10,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:10,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:10,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:10,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:10,889 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:10,889 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:10,889 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:10,889 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:10,889 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:56:10,889 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:10,889 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:10,889 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:10,889 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:56:10,889 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:56:10,890 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:56:10,890 DEBUG [Thread-4043] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:56:10,890 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:56:10,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:56:10,890 DEBUG [Thread-4043] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:56:10,890 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:56:10,890 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:56:10,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:56:10,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(9158): Flush requested on 6873ba93789e63dd9f14391208147267 2023-04-19 20:56:10,931 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 6873ba93789e63dd9f14391208147267 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-19 20:56:10,931 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6873ba93789e63dd9f14391208147267, store=A 2023-04-19 20:56:10,932 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:56:10,932 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6873ba93789e63dd9f14391208147267, store=B 2023-04-19 20:56:10,932 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:56:10,932 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6873ba93789e63dd9f14391208147267, store=C 2023-04-19 20:56:10,932 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:56:10,961 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:10,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57236 deadline: 1681937830957, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:10,961 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:10,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57204 deadline: 1681937830958, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:10,961 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:10,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57226 deadline: 1681937830958, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:10,961 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:10,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57246 deadline: 1681937830958, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:10,962 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:10,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57210 deadline: 1681937830959, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:10,990 DEBUG [Thread-4043] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:56:10,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:56:10,992 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:56:10,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:56:10,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:56:10,994 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:56:10,994 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:56:10,994 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:56:10,994 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:56:10,994 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:10,994 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:10,997 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:10,997 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:56:10,997 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:56:10,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:56:10,998 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:10,998 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:56:10,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:10,998 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:10,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:56:10,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:10,999 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:56:10,999 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:56:10,999 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:56:10,999 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:56:10,999 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:56:10,999 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:56:10,999 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:56:10,999 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:56:10,999 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:10,999 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. started... 2023-04-19 20:56:10,999 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. as already flushing 2023-04-19 20:56:10,999 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:11,000 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:56:11,000 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:56:11,000 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:11,000 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:56:11,000 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:56:11,001 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:11,001 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:11,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:11,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:11,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:11,001 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:11,001 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:56:11,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:11,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:11,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:11,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:11,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:11,003 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:56:11,003 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@27d80bfb[Count = 0] remaining members to acquire global barrier 2023-04-19 20:56:11,003 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:56:11,003 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:11,004 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:11,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:11,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:11,004 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:11,004 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:56:11,004 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:56:11,004 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:56:11,004 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:56:11,006 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:11,006 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:56:11,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:11,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:11,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:11,006 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:56:11,006 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:56:11,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:11,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:11,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:11,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:11,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:11,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:11,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:11,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:56:11,008 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:56:11,008 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:56:11,008 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:56:11,008 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:56:11,008 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:56:11,008 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:56:11,010 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:11,010 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:11,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:11,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:11,010 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:56:11,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:11,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:11,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:11,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:56:11,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:56:11,011 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:11,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:11,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:11,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:11,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:11,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:11,012 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:11,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:11,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:11,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:11,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:11,015 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:11,015 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:11,015 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:11,015 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:11,015 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:56:11,015 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:56:11,015 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:56:11,015 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:11,015 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:11,015 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:56:11,015 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:11,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:56:11,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:56:11,016 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:56:11,016 DEBUG [Thread-4043] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:56:11,016 DEBUG [Thread-4043] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:56:11,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:56:11,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:56:11,065 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:11,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57236 deadline: 1681937831062, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:11,065 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:11,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57226 deadline: 1681937831062, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:11,066 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:11,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57204 deadline: 1681937831062, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:11,066 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:11,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57246 deadline: 1681937831062, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:11,066 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:11,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57210 deadline: 1681937831063, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:11,116 DEBUG [Thread-4043] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:56:11,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:56:11,118 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:56:11,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:56:11,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:56:11,119 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:56:11,119 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:56:11,119 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:56:11,119 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:56:11,120 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:11,120 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:11,121 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:56:11,121 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:11,121 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:56:11,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:56:11,121 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:11,121 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:56:11,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:11,122 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:11,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:56:11,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:11,122 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:56:11,122 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:56:11,123 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:56:11,123 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:56:11,123 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:56:11,123 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:56:11,123 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:56:11,123 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:56:11,123 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:11,123 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. started... 2023-04-19 20:56:11,123 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. as already flushing 2023-04-19 20:56:11,123 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:11,124 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:56:11,124 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:56:11,124 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:11,124 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:56:11,124 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:56:11,125 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:11,126 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:11,126 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:56:11,128 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:11,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:11,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:11,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:11,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:11,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:11,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:11,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:11,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:11,130 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:56:11,130 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@463ae175[Count = 0] remaining members to acquire global barrier 2023-04-19 20:56:11,130 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:56:11,130 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:11,132 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:11,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:11,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:11,132 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:56:11,132 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:56:11,132 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:56:11,132 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:11,132 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:56:11,133 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:11,133 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:56:11,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:11,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:11,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:11,133 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:56:11,134 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:56:11,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:11,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:11,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:11,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:11,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:11,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:11,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:11,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:56:11,136 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:56:11,136 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:56:11,136 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:56:11,136 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:56:11,136 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:56:11,136 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:56:11,137 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:11,137 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:11,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:11,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:11,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:11,138 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:11,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:11,138 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:56:11,138 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:56:11,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:56:11,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:11,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:11,139 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:11,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:11,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:11,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:11,140 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:11,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:11,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:11,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:11,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:11,143 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:11,143 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:11,143 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:56:11,143 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:11,143 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:11,143 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:11,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:56:11,143 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:56:11,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:56:11,143 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:56:11,143 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:56:11,143 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:56:11,143 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:11,143 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:11,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:56:11,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:56:11,143 DEBUG [Thread-4043] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:56:11,143 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:56:11,143 DEBUG [Thread-4043] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:56:11,243 DEBUG [Thread-4043] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:56:11,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:56:11,246 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:56:11,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:56:11,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:56:11,247 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:56:11,247 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:56:11,247 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:56:11,247 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:56:11,248 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:11,248 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:11,250 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:56:11,250 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:11,250 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:56:11,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:56:11,251 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:11,251 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:56:11,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:11,251 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:11,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:56:11,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:11,252 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:56:11,252 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:56:11,252 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:56:11,252 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:56:11,252 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:56:11,252 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:56:11,252 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:56:11,252 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:56:11,252 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:11,252 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. started... 2023-04-19 20:56:11,252 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. as already flushing 2023-04-19 20:56:11,252 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:11,253 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:56:11,253 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:56:11,253 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:11,253 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:56:11,253 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:56:11,257 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:11,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:11,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:11,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:11,257 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:11,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:11,258 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:11,258 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:56:11,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:11,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:11,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:11,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:11,260 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:56:11,260 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6b301c40[Count = 0] remaining members to acquire global barrier 2023-04-19 20:56:11,260 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:56:11,260 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:11,262 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:11,262 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:11,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:11,262 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:56:11,262 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:56:11,262 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:56:11,262 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:11,262 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:56:11,264 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:11,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:11,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:11,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:11,264 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:56:11,264 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:56:11,264 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:56:11,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:11,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:11,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:11,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:11,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:11,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:11,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:11,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:56:11,267 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:56:11,267 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:56:11,267 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:56:11,267 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:56:11,267 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:56:11,267 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:56:11,269 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:11,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57236 deadline: 1681937831266, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:11,269 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:11,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57226 deadline: 1681937831266, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:11,269 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:11,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57204 deadline: 1681937831268, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:11,269 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:11,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57210 deadline: 1681937831268, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:11,269 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:11,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57246 deadline: 1681937831268, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:11,273 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:11,273 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:56:11,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:11,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:11,273 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:11,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:11,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:11,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:11,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:56:11,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:56:11,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:11,273 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:11,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:11,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:11,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:11,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:11,275 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:11,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:11,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:11,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:11,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:11,277 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:56:11,277 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:56:11,278 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:56:11,278 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:56:11,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:56:11,278 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:56:11,278 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:56:11,278 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:11,278 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:11,278 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:11,278 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:11,278 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:11,278 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:11,278 DEBUG [Thread-4043] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:56:11,278 DEBUG [Thread-4043] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:56:11,278 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:56:11,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:56:11,278 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:11,345 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230419c942bf57f1974c0c948bc10c2e39e754 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419c942bf57f1974c0c948bc10c2e39e754 2023-04-19 20:56:11,350 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=117, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/A/38a4c874c52a4ed99810d21c4dee83a5 2023-04-19 20:56:11,378 DEBUG [Thread-4043] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:56:11,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:56:11,380 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:56:11,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:56:11,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:56:11,382 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:56:11,382 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:56:11,382 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:56:11,382 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:56:11,383 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:11,383 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:11,384 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:56:11,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:56:11,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:56:11,384 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:11,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:11,384 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:11,385 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:56:11,385 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:11,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:56:11,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:11,385 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:56:11,385 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:56:11,386 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:56:11,386 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:56:11,386 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:56:11,386 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:56:11,386 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:56:11,386 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:56:11,386 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:11,386 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. started... 2023-04-19 20:56:11,386 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. as already flushing 2023-04-19 20:56:11,386 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:11,386 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:56:11,386 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:56:11,386 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:11,386 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:56:11,386 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:56:11,387 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:11,388 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:11,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:11,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:11,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:11,388 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:11,388 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:56:11,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:11,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:11,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:11,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:11,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:11,389 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:56:11,389 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@61cbf2ab[Count = 0] remaining members to acquire global barrier 2023-04-19 20:56:11,390 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:56:11,390 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:11,391 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:11,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:11,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:11,392 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:56:11,392 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:56:11,392 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:56:11,392 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:11,392 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:56:11,393 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:56:11,393 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:56:11,393 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:11,393 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:56:11,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:11,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:11,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:11,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:11,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:11,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:11,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:11,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:11,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:11,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:11,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:56:11,395 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:56:11,395 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:56:11,395 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:56:11,395 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:56:11,396 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:56:11,396 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:56:11,397 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:11,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:11,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:11,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:11,397 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:11,397 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:56:11,398 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:11,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:11,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:11,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:56:11,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:56:11,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:11,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:11,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:11,399 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:11,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:11,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:11,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:11,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:11,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:11,402 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:11,402 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:11,402 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:11,402 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:11,402 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:11,402 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:11,402 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:11,402 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:56:11,402 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:56:11,402 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:56:11,402 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:56:11,402 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:56:11,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:56:11,402 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:56:11,403 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:56:11,403 DEBUG [Thread-4043] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:56:11,403 DEBUG [Thread-4043] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:56:11,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:56:11,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:56:11,503 DEBUG [Thread-4043] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:56:11,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:56:11,505 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:56:11,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:56:11,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:56:11,506 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:56:11,506 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:56:11,506 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:56:11,506 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:56:11,507 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:11,507 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:11,509 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:11,509 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:56:11,510 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:56:11,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:56:11,510 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:11,510 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:56:11,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:11,510 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:11,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:56:11,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:11,511 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:56:11,511 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:56:11,511 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:56:11,511 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:56:11,511 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:56:11,511 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:56:11,511 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:56:11,511 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:56:11,511 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:11,511 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. started... 2023-04-19 20:56:11,511 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. as already flushing 2023-04-19 20:56:11,511 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:11,512 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:56:11,512 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:56:11,512 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:11,512 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:56:11,512 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:56:11,513 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:11,513 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:11,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:11,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:11,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:11,513 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:11,513 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:56:11,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:11,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:11,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:11,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:11,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:11,515 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:56:11,515 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@62bbaba[Count = 0] remaining members to acquire global barrier 2023-04-19 20:56:11,515 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:56:11,515 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:11,516 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:11,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:11,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:11,516 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:56:11,516 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:56:11,516 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:56:11,516 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:11,516 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:56:11,518 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:11,518 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:56:11,518 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:56:11,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:11,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:11,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:11,518 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:56:11,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:11,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:11,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:11,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:11,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:11,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:11,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:11,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:56:11,520 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:56:11,520 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:56:11,520 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:56:11,520 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:56:11,520 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:56:11,520 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:56:11,522 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:11,522 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:11,522 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:11,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:11,522 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:56:11,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:11,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:11,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:11,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:56:11,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:56:11,523 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:11,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:11,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:11,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:11,524 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:11,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:11,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:11,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:11,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:11,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:11,526 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:56:11,526 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:11,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:56:11,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:56:11,526 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:56:11,526 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:11,527 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:56:11,527 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:11,527 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:11,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:56:11,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:56:11,527 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:56:11,527 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:56:11,527 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:11,527 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:11,527 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:11,527 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:56:11,527 DEBUG [Thread-4043] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:56:11,527 DEBUG [Thread-4043] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:56:11,572 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:11,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57204 deadline: 1681937831570, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:11,572 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:11,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57210 deadline: 1681937831570, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:11,572 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:11,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57226 deadline: 1681937831571, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:11,573 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:11,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57246 deadline: 1681937831571, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:11,573 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:11,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57236 deadline: 1681937831572, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:11,627 DEBUG [Thread-4043] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:56:11,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:56:11,631 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:56:11,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:56:11,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:56:11,632 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:56:11,632 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:56:11,632 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:56:11,632 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:56:11,633 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:11,633 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:11,634 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:11,634 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:56:11,634 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:56:11,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:56:11,634 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:11,635 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:56:11,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:11,635 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:11,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:56:11,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:11,635 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:56:11,636 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:56:11,636 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:56:11,636 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:56:11,636 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:56:11,636 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:56:11,636 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:56:11,636 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:56:11,636 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:11,636 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. started... 2023-04-19 20:56:11,636 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. as already flushing 2023-04-19 20:56:11,636 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:11,636 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:56:11,636 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:56:11,636 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:11,637 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:56:11,637 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:56:11,638 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:11,638 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:11,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:11,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:11,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:11,638 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:11,639 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:56:11,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:11,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:11,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:11,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:11,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:11,640 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:56:11,640 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@170bd243[Count = 0] remaining members to acquire global barrier 2023-04-19 20:56:11,640 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:56:11,640 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:11,645 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:11,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:11,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:11,646 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:56:11,646 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:56:11,646 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:56:11,646 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:11,646 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:56:11,647 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:11,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:11,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:11,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:11,647 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:56:11,647 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:56:11,647 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:56:11,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:11,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:11,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:11,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:11,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:11,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:11,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:11,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:56:11,650 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:56:11,650 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:56:11,650 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:56:11,650 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:56:11,650 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:56:11,650 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:56:11,651 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:11,651 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:11,651 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:56:11,651 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:11,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:11,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:11,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:11,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:11,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:56:11,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:56:11,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:11,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:11,652 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:11,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:11,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:11,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:11,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:11,653 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:11,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:11,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:11,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:11,656 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:11,656 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:56:11,657 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:56:11,657 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:56:11,657 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:56:11,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:56:11,657 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:56:11,657 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:11,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:56:11,657 DEBUG [Thread-4043] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:56:11,657 DEBUG [Thread-4043] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:56:11,657 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:11,657 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:11,657 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:11,657 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:11,657 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:11,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:56:11,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:56:11,757 DEBUG [Thread-4043] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:56:11,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:56:11,759 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:56:11,760 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=117 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/B/a944e8d265374600957b235c230ff704 2023-04-19 20:56:11,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:56:11,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:56:11,761 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:56:11,761 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:56:11,761 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:56:11,761 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:56:11,762 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:11,762 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:11,763 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:56:11,763 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:11,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:56:11,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:56:11,763 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:11,763 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:56:11,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:11,764 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:11,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:56:11,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:11,764 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:56:11,765 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:56:11,765 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:56:11,765 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:56:11,765 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:56:11,765 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:56:11,765 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:56:11,765 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:56:11,765 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:11,765 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. started... 2023-04-19 20:56:11,765 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. as already flushing 2023-04-19 20:56:11,766 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:11,766 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:56:11,766 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:56:11,766 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:11,766 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:56:11,766 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:56:11,768 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:11,768 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:11,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:11,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:11,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:11,768 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:11,769 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:56:11,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:11,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:11,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:11,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:11,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:11,771 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:56:11,771 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7cb20efe[Count = 0] remaining members to acquire global barrier 2023-04-19 20:56:11,771 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:56:11,771 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:11,772 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=117 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/C/a5d417ff25964628a8b91bef78c15d8b 2023-04-19 20:56:11,772 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:11,772 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:11,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:11,772 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:56:11,772 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:56:11,772 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:56:11,773 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:11,773 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:56:11,776 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/A/38a4c874c52a4ed99810d21c4dee83a5 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/38a4c874c52a4ed99810d21c4dee83a5 2023-04-19 20:56:11,776 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:56:11,776 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:56:11,776 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:56:11,776 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:11,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:11,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:11,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:11,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:11,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:11,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:11,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:11,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:11,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:11,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:11,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:56:11,779 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:56:11,779 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:56:11,779 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:56:11,779 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:56:11,779 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:56:11,779 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:56:11,781 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/38a4c874c52a4ed99810d21c4dee83a5, entries=300, sequenceid=117, filesize=45.5 K 2023-04-19 20:56:11,781 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:11,781 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:11,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:11,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:11,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:11,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:11,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:11,781 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:56:11,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:56:11,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:56:11,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:11,782 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/B/a944e8d265374600957b235c230ff704 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/a944e8d265374600957b235c230ff704 2023-04-19 20:56:11,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:11,783 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:11,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:11,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:11,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:11,784 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:11,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:11,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:11,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:11,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:11,786 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/a944e8d265374600957b235c230ff704, entries=100, sequenceid=117, filesize=9.3 K 2023-04-19 20:56:11,787 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:11,787 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:11,787 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:11,787 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:56:11,787 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:56:11,787 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:56:11,787 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:56:11,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:56:11,787 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:56:11,787 DEBUG [Thread-4043] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:56:11,787 DEBUG [Thread-4043] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:56:11,787 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:56:11,787 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:56:11,787 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:11,787 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:56:11,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:56:11,788 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:11,788 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/C/a5d417ff25964628a8b91bef78c15d8b as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/a5d417ff25964628a8b91bef78c15d8b 2023-04-19 20:56:11,788 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:11,788 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:11,791 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/a5d417ff25964628a8b91bef78c15d8b, entries=100, sequenceid=117, filesize=9.3 K 2023-04-19 20:56:11,792 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 6873ba93789e63dd9f14391208147267 in 861ms, sequenceid=117, compaction requested=true 2023-04-19 20:56:11,792 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 6873ba93789e63dd9f14391208147267: 2023-04-19 20:56:11,792 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 20:56:11,792 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 20:56:11,792 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-19 20:56:11,792 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-19 20:56:11,792 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 20:56:11,793 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 57189 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-19 20:56:11,793 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1912): 6873ba93789e63dd9f14391208147267/B is initiating minor compaction (all files) 2023-04-19 20:56:11,793 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 149653 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-19 20:56:11,793 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6873ba93789e63dd9f14391208147267/B in TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:11,794 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1912): 6873ba93789e63dd9f14391208147267/A is initiating minor compaction (all files) 2023-04-19 20:56:11,794 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/d3ae7695e34940de99097e297a7035ae, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/d0eeacf0aa3f452b912d0c4cf3a29642, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/2d48a2781f9b434688b5d141f60c78a2, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/9e60c267bda7478ba55d83e91c3da20b, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/a944e8d265374600957b235c230ff704] into tmpdir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp, totalSize=55.8 K 2023-04-19 20:56:11,794 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 6873ba93789e63dd9f14391208147267/A in TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:11,794 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/eec6a4efa75342c8aaa239f120fc5773, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/8f95ea25964a4a5cb50534c22f1f2dfb, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/f8f1c07250574ed481e0dca56a8de79c, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/3fa83711bf924f2f80a381c4fa673d33, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/38a4c874c52a4ed99810d21c4dee83a5] into tmpdir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp, totalSize=146.1 K 2023-04-19 20:56:11,794 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting d3ae7695e34940de99097e297a7035ae, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=52, earliestPutTs=1681937767175 2023-04-19 20:56:11,794 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting d0eeacf0aa3f452b912d0c4cf3a29642, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=68, earliestPutTs=1681937767498 2023-04-19 20:56:11,794 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting eec6a4efa75342c8aaa239f120fc5773, keycount=150, bloomtype=ROW, size=25.2 K, encoding=NONE, compression=NONE, seqNum=52, earliestPutTs=1681937767175 2023-04-19 20:56:11,795 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 2d48a2781f9b434688b5d141f60c78a2, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=87, earliestPutTs=1681937768119 2023-04-19 20:56:11,795 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 8f95ea25964a4a5cb50534c22f1f2dfb, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=68, earliestPutTs=1681937767498 2023-04-19 20:56:11,795 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 9e60c267bda7478ba55d83e91c3da20b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=105, earliestPutTs=1681937768767 2023-04-19 20:56:11,795 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting f8f1c07250574ed481e0dca56a8de79c, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=87, earliestPutTs=1681937768119 2023-04-19 20:56:11,796 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting a944e8d265374600957b235c230ff704, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=117, earliestPutTs=1681937770929 2023-04-19 20:56:11,796 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 3fa83711bf924f2f80a381c4fa673d33, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=105, earliestPutTs=1681937768767 2023-04-19 20:56:11,796 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 38a4c874c52a4ed99810d21c4dee83a5, keycount=300, bloomtype=ROW, size=45.5 K, encoding=NONE, compression=NONE, seqNum=117, earliestPutTs=1681937770910 2023-04-19 20:56:11,805 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] throttle.PressureAwareThroughputController(145): 6873ba93789e63dd9f14391208147267#B#compaction#568 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-19 20:56:11,808 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] throttle.PressureAwareThroughputController(145): 6873ba93789e63dd9f14391208147267#A#compaction#567 average throughput is 4.93 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 20:56:11,887 DEBUG [Thread-4043] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:56:11,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:56:11,890 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:56:11,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:56:11,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:56:11,891 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:56:11,891 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:56:11,891 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:56:11,891 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:56:11,891 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:11,891 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:11,892 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:11,892 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:56:11,893 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:56:11,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:56:11,893 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:11,893 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:56:11,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:11,893 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:11,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:56:11,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:11,894 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:56:11,894 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:56:11,894 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:56:11,894 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:56:11,894 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:56:11,894 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:56:11,894 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:56:11,894 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:56:11,894 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:11,894 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. started... 2023-04-19 20:56:11,894 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6873ba93789e63dd9f14391208147267 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-19 20:56:11,895 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6873ba93789e63dd9f14391208147267, store=A 2023-04-19 20:56:11,895 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:56:11,895 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6873ba93789e63dd9f14391208147267, store=B 2023-04-19 20:56:11,895 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:56:11,895 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6873ba93789e63dd9f14391208147267, store=C 2023-04-19 20:56:11,895 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:56:11,931 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230419e62b7ddf153b413dbf7a444d0ed8bc03 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419e62b7ddf153b413dbf7a444d0ed8bc03 2023-04-19 20:56:11,941 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=137, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/A/524ff811bb4f44098edc0857a6061a60 2023-04-19 20:56:11,955 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=137 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/B/83399b14dae74fc2a37a1b5c1f80e84e 2023-04-19 20:56:11,964 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=137 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/C/e3fddbd147a44fb89c13661b67f3f832 2023-04-19 20:56:11,967 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/A/524ff811bb4f44098edc0857a6061a60 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/524ff811bb4f44098edc0857a6061a60 2023-04-19 20:56:11,971 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/524ff811bb4f44098edc0857a6061a60, entries=150, sequenceid=137, filesize=25.3 K 2023-04-19 20:56:11,972 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/B/83399b14dae74fc2a37a1b5c1f80e84e as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/83399b14dae74fc2a37a1b5c1f80e84e 2023-04-19 20:56:11,976 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/83399b14dae74fc2a37a1b5c1f80e84e, entries=150, sequenceid=137, filesize=11.8 K 2023-04-19 20:56:11,976 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/C/e3fddbd147a44fb89c13661b67f3f832 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/e3fddbd147a44fb89c13661b67f3f832 2023-04-19 20:56:11,979 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/e3fddbd147a44fb89c13661b67f3f832, entries=150, sequenceid=137, filesize=11.8 K 2023-04-19 20:56:11,980 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=0 B/0 for 6873ba93789e63dd9f14391208147267 in 86ms, sequenceid=137, compaction requested=true 2023-04-19 20:56:11,980 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6873ba93789e63dd9f14391208147267: 2023-04-19 20:56:11,980 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:11,980 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:56:11,981 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:56:11,981 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:11,981 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:56:11,981 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:56:11,984 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:11,984 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:11,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:11,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:11,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:11,984 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:11,984 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:56:11,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:11,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:11,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:11,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:11,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:11,986 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:56:11,986 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4252e569[Count = 0] remaining members to acquire global barrier 2023-04-19 20:56:11,986 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:56:11,986 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:11,987 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:11,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:11,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:11,987 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:56:11,987 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:56:11,987 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:56:11,988 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:11,988 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:56:11,989 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:56:11,989 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:56:11,989 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:11,989 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:56:11,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:11,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:11,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:11,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:11,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:11,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:11,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:11,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:11,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:11,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:11,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:56:11,992 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:56:11,992 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:56:11,992 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:56:11,992 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:56:11,992 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:56:11,992 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:56:11,993 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:11,993 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:11,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:11,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:11,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:11,994 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:56:11,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:11,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:11,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:56:11,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:56:11,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:11,994 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:11,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:11,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:11,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:11,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:11,996 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:11,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:11,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:11,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:11,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:11,998 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:11,998 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:11,998 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:11,999 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:56:11,998 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:56:11,999 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:56:11,999 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:56:11,999 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:11,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:56:11,999 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:56:11,999 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:56:11,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:56:11,999 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:11,999 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:11,999 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:11,999 DEBUG [Thread-4043] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:56:11,999 DEBUG [Thread-4043] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:56:11,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:56:11,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:56:12,099 DEBUG [Thread-4043] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:56:12,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:56:12,101 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:56:12,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(9158): Flush requested on 6873ba93789e63dd9f14391208147267 2023-04-19 20:56:12,102 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 6873ba93789e63dd9f14391208147267 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-19 20:56:12,102 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6873ba93789e63dd9f14391208147267, store=A 2023-04-19 20:56:12,102 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:56:12,102 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6873ba93789e63dd9f14391208147267, store=B 2023-04-19 20:56:12,102 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:56:12,102 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6873ba93789e63dd9f14391208147267, store=C 2023-04-19 20:56:12,102 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:56:12,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:56:12,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:56:12,105 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:56:12,105 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:56:12,105 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:56:12,105 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:56:12,105 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:12,105 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:12,107 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:56:12,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:56:12,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:56:12,107 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:12,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:12,107 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:12,108 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:56:12,108 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:12,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:56:12,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:12,108 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:56:12,110 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:56:12,110 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:56:12,110 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:56:12,110 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:56:12,110 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:56:12,111 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:56:12,111 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:56:12,111 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:12,111 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. started... 2023-04-19 20:56:12,111 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. as already flushing 2023-04-19 20:56:12,111 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:12,111 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:56:12,111 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:56:12,111 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:12,111 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:56:12,111 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:56:12,113 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:12,113 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:12,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:12,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:12,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:12,114 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:12,114 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:56:12,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:12,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:12,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:12,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:12,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:12,115 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:56:12,115 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@197779f3[Count = 0] remaining members to acquire global barrier 2023-04-19 20:56:12,115 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:56:12,115 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:12,117 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:12,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:12,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:12,117 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:56:12,117 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:56:12,117 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:56:12,117 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:12,117 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:56:12,119 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:56:12,119 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:56:12,120 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:56:12,120 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:12,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:12,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:12,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:12,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:12,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:12,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:12,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:12,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:12,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:12,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:12,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:56:12,123 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:56:12,123 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:56:12,123 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:56:12,123 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:56:12,123 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:56:12,123 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:56:12,127 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:12,127 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:12,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:12,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:12,127 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:56:12,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:12,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:12,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:12,128 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:56:12,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:56:12,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:12,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:12,128 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:12,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:12,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:12,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:12,130 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:12,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:12,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:12,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:12,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:12,134 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:56:12,134 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:56:12,134 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:56:12,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:56:12,134 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:12,135 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:12,135 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:12,135 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:12,135 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:12,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:56:12,135 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:56:12,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:56:12,135 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:56:12,135 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:56:12,135 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:56:12,135 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:12,135 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:12,135 DEBUG [Thread-4043] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:56:12,135 DEBUG [Thread-4043] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:56:12,145 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681937772129, max timestamp=1681937772142Num uniques 859; 2023-04-19 20:56:12,145 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-19 20:56:12,146 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681937772129, max timestamp=1681937772144Num uniques 859; 2023-04-19 20:56:12,146 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-19 20:56:12,146 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681937772129, max timestamp=1681937772144Num uniques 859; 2023-04-19 20:56:12,146 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-19 20:56:12,154 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304195e9931f9c02248b4b523ad3370cfc4c0 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304195e9931f9c02248b4b523ad3370cfc4c0 2023-04-19 20:56:12,178 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:12,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57210 deadline: 1681937832168, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:12,178 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:12,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57204 deadline: 1681937832169, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:12,178 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:12,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57236 deadline: 1681937832170, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:12,179 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:12,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57246 deadline: 1681937832173, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:12,183 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:12,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57226 deadline: 1681937832178, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:12,217 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/B/9acfcde5fd774be59026248e71165777 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/9acfcde5fd774be59026248e71165777 2023-04-19 20:56:12,221 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 6873ba93789e63dd9f14391208147267/B of 6873ba93789e63dd9f14391208147267 into 9acfcde5fd774be59026248e71165777(size=11.9 K), total size for store is 23.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 20:56:12,221 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6873ba93789e63dd9f14391208147267: 2023-04-19 20:56:12,221 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267., storeName=6873ba93789e63dd9f14391208147267/B, priority=11, startTime=1681937771792; duration=0sec 2023-04-19 20:56:12,222 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 20:56:12,222 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-19 20:56:12,224 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 69225 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-19 20:56:12,225 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1912): 6873ba93789e63dd9f14391208147267/C is initiating minor compaction (all files) 2023-04-19 20:56:12,225 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6873ba93789e63dd9f14391208147267/C in TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:12,225 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/459f086d2f8a4a988a5447fbb85d6d74, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/ec555bd2579447849138ddb079eeef00, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/66ec2c0da0ab4ddfb833b817a8649e63, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/5c6e274922f44ba7aa8a69aced8334cb, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/a5d417ff25964628a8b91bef78c15d8b, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/e3fddbd147a44fb89c13661b67f3f832] into tmpdir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp, totalSize=67.6 K 2023-04-19 20:56:12,225 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 459f086d2f8a4a988a5447fbb85d6d74, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=52, earliestPutTs=1681937767175 2023-04-19 20:56:12,225 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting ec555bd2579447849138ddb079eeef00, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=68, earliestPutTs=1681937767498 2023-04-19 20:56:12,226 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 66ec2c0da0ab4ddfb833b817a8649e63, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=87, earliestPutTs=1681937768119 2023-04-19 20:56:12,226 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 5c6e274922f44ba7aa8a69aced8334cb, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=105, earliestPutTs=1681937768767 2023-04-19 20:56:12,227 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting a5d417ff25964628a8b91bef78c15d8b, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=117, earliestPutTs=1681937770929 2023-04-19 20:56:12,227 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting e3fddbd147a44fb89c13661b67f3f832, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=137, earliestPutTs=1681937770956 2023-04-19 20:56:12,233 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/A/c8a7518ef45b49388d22210fd2f92996 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/c8a7518ef45b49388d22210fd2f92996 2023-04-19 20:56:12,235 DEBUG [Thread-4043] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:56:12,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:56:12,238 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 6873ba93789e63dd9f14391208147267/A of 6873ba93789e63dd9f14391208147267 into c8a7518ef45b49388d22210fd2f92996(size=25.4 K), total size for store is 50.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 20:56:12,238 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6873ba93789e63dd9f14391208147267: 2023-04-19 20:56:12,238 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267., storeName=6873ba93789e63dd9f14391208147267/A, priority=11, startTime=1681937771792; duration=0sec 2023-04-19 20:56:12,238 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 20:56:12,238 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:56:12,239 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] throttle.PressureAwareThroughputController(145): 6873ba93789e63dd9f14391208147267#C#compaction#573 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 20:56:12,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:56:12,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:56:12,240 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:56:12,240 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:56:12,240 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:56:12,240 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:56:12,240 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:12,240 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:12,242 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:12,242 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:56:12,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:56:12,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:56:12,242 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:12,242 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:56:12,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:12,243 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:12,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:56:12,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:12,243 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:56:12,243 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:56:12,243 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:56:12,243 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:56:12,243 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:56:12,243 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:56:12,248 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:56:12,248 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:56:12,248 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:12,248 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. started... 2023-04-19 20:56:12,248 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. as already flushing 2023-04-19 20:56:12,248 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:12,248 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:56:12,248 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:56:12,248 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:12,248 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:56:12,248 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:56:12,252 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/C/c55e6ce76d53470f9a91643a0fa72029 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/c55e6ce76d53470f9a91643a0fa72029 2023-04-19 20:56:12,255 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:12,255 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:12,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:12,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:12,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:12,255 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 6873ba93789e63dd9f14391208147267/C of 6873ba93789e63dd9f14391208147267 into c55e6ce76d53470f9a91643a0fa72029(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 20:56:12,256 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:12,256 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:56:12,256 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6873ba93789e63dd9f14391208147267: 2023-04-19 20:56:12,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:12,256 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267., storeName=6873ba93789e63dd9f14391208147267/C, priority=10, startTime=1681937771792; duration=0sec 2023-04-19 20:56:12,256 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 20:56:12,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:12,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:12,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:12,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:12,257 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:56:12,258 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@50273954[Count = 0] remaining members to acquire global barrier 2023-04-19 20:56:12,258 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:56:12,258 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:12,259 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:12,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:12,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:12,259 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:56:12,259 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:56:12,259 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:56:12,259 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:12,259 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:56:12,260 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:56:12,260 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:56:12,260 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:12,260 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:56:12,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:12,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:12,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:12,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:12,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:12,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:12,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:12,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:12,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:12,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:12,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:56:12,265 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:56:12,265 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:56:12,265 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:56:12,265 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:56:12,265 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:56:12,265 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:56:12,267 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:12,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:12,267 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:12,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:12,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:12,267 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:12,267 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:56:12,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:12,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:12,267 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:56:12,267 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:12,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:56:12,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:12,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:12,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:12,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:12,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:12,269 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:12,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:12,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:12,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:12,271 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:12,272 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:12,272 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:12,272 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:56:12,272 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:12,272 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:56:12,272 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:12,272 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:56:12,272 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:56:12,272 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:56:12,272 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:56:12,272 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:56:12,272 DEBUG [Thread-4043] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:56:12,272 DEBUG [Thread-4043] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:56:12,272 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:12,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:56:12,272 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:12,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:56:12,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:56:12,281 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:12,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57210 deadline: 1681937832279, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:12,282 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:12,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57246 deadline: 1681937832281, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:12,285 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:12,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57226 deadline: 1681937832283, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:12,285 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:12,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57204 deadline: 1681937832284, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:12,285 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:12,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57236 deadline: 1681937832285, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:12,372 DEBUG [Thread-4043] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:56:12,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:56:12,374 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:56:12,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:56:12,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:56:12,375 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:56:12,375 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:56:12,375 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:56:12,375 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:56:12,376 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:12,376 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:12,377 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:56:12,377 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:12,377 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:56:12,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:56:12,377 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:12,377 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:56:12,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:12,378 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:12,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:56:12,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:12,378 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:56:12,378 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:56:12,378 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:56:12,378 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:56:12,379 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:56:12,379 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:56:12,379 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:56:12,379 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:56:12,379 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:12,379 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. started... 2023-04-19 20:56:12,379 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. as already flushing 2023-04-19 20:56:12,379 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:12,379 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:56:12,379 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:56:12,379 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:12,379 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:56:12,379 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:56:12,380 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:12,381 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:12,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:12,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:12,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:12,381 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:12,381 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:56:12,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:12,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:12,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:12,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:12,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:12,382 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:56:12,382 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3c8e0850[Count = 0] remaining members to acquire global barrier 2023-04-19 20:56:12,382 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:56:12,382 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:12,383 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:12,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:12,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:12,384 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:56:12,384 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:12,384 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:56:12,384 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:56:12,384 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:56:12,386 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:56:12,386 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:56:12,386 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:12,387 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:56:12,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:12,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:12,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:12,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:12,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:12,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:12,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:12,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:12,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:12,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:12,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:56:12,389 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:56:12,389 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:56:12,389 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:56:12,389 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:56:12,389 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:56:12,389 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:56:12,392 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:12,392 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:12,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:12,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:12,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:12,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:12,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:12,392 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:56:12,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:12,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:56:12,392 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:12,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:56:12,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:12,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:12,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:12,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:12,394 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:12,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:12,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:12,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:12,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:12,396 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:56:12,396 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:56:12,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:56:12,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:56:12,396 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:12,397 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:12,397 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:12,397 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:12,397 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:12,397 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:12,397 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:12,397 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:56:12,397 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:56:12,397 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:56:12,397 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:56:12,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:56:12,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:56:12,397 DEBUG [Thread-4043] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:56:12,397 DEBUG [Thread-4043] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:56:12,485 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:12,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57210 deadline: 1681937832484, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:12,485 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:12,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57246 deadline: 1681937832484, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:12,488 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:12,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57226 deadline: 1681937832486, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:12,489 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:12,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57204 deadline: 1681937832487, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:12,489 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:12,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57236 deadline: 1681937832487, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:12,497 DEBUG [Thread-4043] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:56:12,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:56:12,499 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:56:12,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:56:12,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:56:12,500 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:56:12,500 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:56:12,500 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:56:12,500 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:56:12,501 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:12,501 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:12,502 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:12,502 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:56:12,502 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:56:12,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:56:12,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:12,502 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:12,503 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:56:12,503 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:12,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:56:12,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:12,503 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:56:12,503 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:56:12,503 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:56:12,503 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:56:12,503 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:56:12,503 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:56:12,504 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:56:12,504 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:56:12,504 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:12,504 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. started... 2023-04-19 20:56:12,504 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. as already flushing 2023-04-19 20:56:12,504 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:12,504 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:56:12,504 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:56:12,504 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:12,504 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:56:12,504 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:56:12,506 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:12,506 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:12,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:12,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:12,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:12,506 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:12,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:12,506 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:56:12,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:12,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:12,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:12,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:12,508 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:56:12,508 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6922db05[Count = 0] remaining members to acquire global barrier 2023-04-19 20:56:12,508 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:56:12,508 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:12,509 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:12,509 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:12,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:12,509 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:56:12,509 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:56:12,509 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:56:12,510 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:12,510 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:56:12,511 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:12,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:12,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:12,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:12,511 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:56:12,511 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:56:12,511 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:56:12,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:12,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:12,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:12,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:12,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:12,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:12,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:12,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:56:12,514 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:56:12,514 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:56:12,514 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:56:12,514 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:56:12,514 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:56:12,514 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:56:12,516 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:12,516 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:12,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:12,516 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:56:12,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:12,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:12,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:12,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:12,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:12,516 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:12,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:56:12,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:56:12,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:12,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:12,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:12,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:12,518 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:12,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:12,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:12,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:12,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:12,523 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:12,523 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:12,523 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:12,523 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:12,523 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:56:12,523 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:56:12,523 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:56:12,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:56:12,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:56:12,523 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:56:12,523 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:56:12,523 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:56:12,523 DEBUG [Thread-4043] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:56:12,524 DEBUG [Thread-4043] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:56:12,523 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:12,524 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:12,524 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:12,524 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:56:12,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:56:12,578 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=147, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/A/ddc21f4dd5f64c789829525f787ae63b 2023-04-19 20:56:12,587 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=147 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/B/944cfc07c5b746fc98b8c2af008c2732 2023-04-19 20:56:12,594 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=147 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/C/17dd83e3da5247d9b0da715ea48bda24 2023-04-19 20:56:12,597 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/A/ddc21f4dd5f64c789829525f787ae63b as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/ddc21f4dd5f64c789829525f787ae63b 2023-04-19 20:56:12,600 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/ddc21f4dd5f64c789829525f787ae63b, entries=350, sequenceid=147, filesize=52.6 K 2023-04-19 20:56:12,600 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/B/944cfc07c5b746fc98b8c2af008c2732 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/944cfc07c5b746fc98b8c2af008c2732 2023-04-19 20:56:12,603 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/944cfc07c5b746fc98b8c2af008c2732, entries=150, sequenceid=147, filesize=11.8 K 2023-04-19 20:56:12,604 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/C/17dd83e3da5247d9b0da715ea48bda24 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/17dd83e3da5247d9b0da715ea48bda24 2023-04-19 20:56:12,607 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/17dd83e3da5247d9b0da715ea48bda24, entries=150, sequenceid=147, filesize=11.8 K 2023-04-19 20:56:12,607 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 6873ba93789e63dd9f14391208147267 in 505ms, sequenceid=147, compaction requested=true 2023-04-19 20:56:12,607 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 6873ba93789e63dd9f14391208147267: 2023-04-19 20:56:12,607 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 20:56:12,608 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-19 20:56:12,608 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-19 20:56:12,608 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 20:56:12,608 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 20:56:12,609 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 105776 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-19 20:56:12,609 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1912): 6873ba93789e63dd9f14391208147267/A is initiating minor compaction (all files) 2023-04-19 20:56:12,609 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36232 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-19 20:56:12,609 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 6873ba93789e63dd9f14391208147267/A in TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:12,609 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1912): 6873ba93789e63dd9f14391208147267/B is initiating minor compaction (all files) 2023-04-19 20:56:12,609 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/c8a7518ef45b49388d22210fd2f92996, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/524ff811bb4f44098edc0857a6061a60, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/ddc21f4dd5f64c789829525f787ae63b] into tmpdir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp, totalSize=103.3 K 2023-04-19 20:56:12,609 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6873ba93789e63dd9f14391208147267/B in TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:12,609 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/9acfcde5fd774be59026248e71165777, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/83399b14dae74fc2a37a1b5c1f80e84e, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/944cfc07c5b746fc98b8c2af008c2732] into tmpdir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp, totalSize=35.4 K 2023-04-19 20:56:12,610 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting c8a7518ef45b49388d22210fd2f92996, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=117, earliestPutTs=1681937768767 2023-04-19 20:56:12,610 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 9acfcde5fd774be59026248e71165777, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=117, earliestPutTs=1681937768767 2023-04-19 20:56:12,610 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 524ff811bb4f44098edc0857a6061a60, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=137, earliestPutTs=1681937770956 2023-04-19 20:56:12,610 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 83399b14dae74fc2a37a1b5c1f80e84e, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=137, earliestPutTs=1681937770956 2023-04-19 20:56:12,611 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting ddc21f4dd5f64c789829525f787ae63b, keycount=350, bloomtype=ROW, size=52.6 K, encoding=NONE, compression=NONE, seqNum=147, earliestPutTs=1681937772092 2023-04-19 20:56:12,611 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 944cfc07c5b746fc98b8c2af008c2732, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=147, earliestPutTs=1681937772097 2023-04-19 20:56:12,618 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] throttle.PressureAwareThroughputController(145): 6873ba93789e63dd9f14391208147267#B#compaction#576 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 20:56:12,624 DEBUG [Thread-4043] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:56:12,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:56:12,626 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:56:12,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:56:12,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:56:12,628 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:56:12,628 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:56:12,628 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:56:12,628 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:56:12,628 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] throttle.PressureAwareThroughputController(145): 6873ba93789e63dd9f14391208147267#A#compaction#577 average throughput is 2.46 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 20:56:12,628 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:12,628 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:12,630 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:56:12,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:56:12,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:56:12,630 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:12,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:12,630 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:12,630 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:56:12,631 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:12,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:56:12,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:12,631 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:56:12,631 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:56:12,631 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:56:12,631 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:56:12,631 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:56:12,631 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:56:12,632 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:56:12,632 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:56:12,632 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:12,632 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. started... 2023-04-19 20:56:12,632 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6873ba93789e63dd9f14391208147267 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-04-19 20:56:12,633 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6873ba93789e63dd9f14391208147267, store=A 2023-04-19 20:56:12,633 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 20:56:12,633 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6873ba93789e63dd9f14391208147267, store=B 2023-04-19 20:56:12,633 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 20:56:12,633 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6873ba93789e63dd9f14391208147267, store=C 2023-04-19 20:56:12,633 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 20:56:12,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(9158): Flush requested on 6873ba93789e63dd9f14391208147267 2023-04-19 20:56:12,788 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. as already flushing 2023-04-19 20:56:12,816 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:12,816 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:12,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57210 deadline: 1681937832793, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:12,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57236 deadline: 1681937832794, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:12,821 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:12,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57246 deadline: 1681937832816, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:12,821 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:12,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57204 deadline: 1681937832816, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:12,822 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:12,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57226 deadline: 1681937832816, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:12,919 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:12,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57236 deadline: 1681937832917, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:12,919 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:12,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57210 deadline: 1681937832917, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:12,924 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:12,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57246 deadline: 1681937832922, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:12,924 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:12,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57204 deadline: 1681937832922, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:12,925 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:12,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57226 deadline: 1681937832923, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:13,030 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/B/eb215aa8e54e4924910d390c3e57fc28 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/eb215aa8e54e4924910d390c3e57fc28 2023-04-19 20:56:13,034 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 6873ba93789e63dd9f14391208147267/B of 6873ba93789e63dd9f14391208147267 into eb215aa8e54e4924910d390c3e57fc28(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 20:56:13,034 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6873ba93789e63dd9f14391208147267: 2023-04-19 20:56:13,035 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267., storeName=6873ba93789e63dd9f14391208147267/B, priority=13, startTime=1681937772608; duration=0sec 2023-04-19 20:56:13,035 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 20:56:13,035 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-04-19 20:56:13,035 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-19 20:56:13,036 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-19 20:56:13,036 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. because compaction request was cancelled 2023-04-19 20:56:13,047 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/A/350e37412298417db2ae5e7654af6c20 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/350e37412298417db2ae5e7654af6c20 2023-04-19 20:56:13,051 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 6873ba93789e63dd9f14391208147267/A of 6873ba93789e63dd9f14391208147267 into 350e37412298417db2ae5e7654af6c20(size=25.7 K), total size for store is 25.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 20:56:13,051 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6873ba93789e63dd9f14391208147267: 2023-04-19 20:56:13,051 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267., storeName=6873ba93789e63dd9f14391208147267/A, priority=13, startTime=1681937772607; duration=0sec 2023-04-19 20:56:13,051 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 20:56:13,054 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041965b3241125eb4616a370b96cb35a202a to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041965b3241125eb4616a370b96cb35a202a 2023-04-19 20:56:13,122 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:13,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57236 deadline: 1681937833121, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:13,123 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:13,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57210 deadline: 1681937833121, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:13,126 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:13,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57204 deadline: 1681937833125, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:13,128 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:13,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57246 deadline: 1681937833126, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:13,128 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:13,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57226 deadline: 1681937833127, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:13,427 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:13,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57236 deadline: 1681937833425, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:13,427 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:13,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57210 deadline: 1681937833426, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:13,428 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:13,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57204 deadline: 1681937833427, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:13,430 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:13,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57226 deadline: 1681937833429, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:13,433 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:13,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57246 deadline: 1681937833431, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:13,460 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=186, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/A/ccce7b41508b4527bdf147853f2446dc 2023-04-19 20:56:13,481 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=186 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/B/12ac88195ff64134a627c06b25f6658c 2023-04-19 20:56:13,498 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=186 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/C/3fcf93729c9f44ebbb082fda90a8d510 2023-04-19 20:56:13,502 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/A/ccce7b41508b4527bdf147853f2446dc as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/ccce7b41508b4527bdf147853f2446dc 2023-04-19 20:56:13,505 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/ccce7b41508b4527bdf147853f2446dc, entries=150, sequenceid=186, filesize=25.3 K 2023-04-19 20:56:13,506 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/B/12ac88195ff64134a627c06b25f6658c as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/12ac88195ff64134a627c06b25f6658c 2023-04-19 20:56:13,510 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/12ac88195ff64134a627c06b25f6658c, entries=150, sequenceid=186, filesize=11.8 K 2023-04-19 20:56:13,511 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/C/3fcf93729c9f44ebbb082fda90a8d510 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/3fcf93729c9f44ebbb082fda90a8d510 2023-04-19 20:56:13,515 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/3fcf93729c9f44ebbb082fda90a8d510, entries=150, sequenceid=186, filesize=11.8 K 2023-04-19 20:56:13,515 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=53.67 KB/54960 for 6873ba93789e63dd9f14391208147267 in 883ms, sequenceid=186, compaction requested=true 2023-04-19 20:56:13,516 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6873ba93789e63dd9f14391208147267: 2023-04-19 20:56:13,516 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:13,516 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:56:13,516 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:56:13,516 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:13,516 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:56:13,516 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:56:13,518 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:13,518 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:13,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:13,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:13,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:13,518 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:13,518 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:56:13,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:13,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:13,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:13,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:13,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:13,519 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:56:13,519 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4bb89f33[Count = 0] remaining members to acquire global barrier 2023-04-19 20:56:13,519 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:56:13,519 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:13,520 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:13,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:13,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:13,521 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:56:13,521 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:56:13,521 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:56:13,521 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:13,521 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:56:13,523 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:13,523 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:56:13,523 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:56:13,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:13,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:13,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:13,523 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:56:13,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:13,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:13,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:13,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:13,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:13,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:13,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:13,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:56:13,526 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:56:13,526 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:56:13,526 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:56:13,526 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:56:13,526 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:56:13,526 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:56:13,528 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:13,529 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:56:13,529 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:13,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:13,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:13,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:13,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:13,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:13,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:56:13,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:56:13,529 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:13,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:13,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:13,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:13,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:13,530 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:13,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:13,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:13,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:13,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:13,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:13,533 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:13,533 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:56:13,533 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:56:13,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:56:13,533 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:56:13,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:56:13,533 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:56:13,533 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:13,533 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:13,533 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:13,533 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:13,533 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:13,533 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:13,533 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:56:13,533 DEBUG [Thread-4043] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:56:13,534 DEBUG [Thread-4043] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:56:13,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:56:13,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:56:13,534 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:56:13,634 DEBUG [Thread-4043] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:56:13,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:56:13,636 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:56:13,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:56:13,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:56:13,637 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:56:13,637 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:56:13,637 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:56:13,637 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:56:13,638 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:13,638 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:13,639 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:13,639 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:56:13,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:56:13,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:56:13,639 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:13,639 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:56:13,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:13,640 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:13,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:56:13,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:13,641 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:56:13,641 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:56:13,641 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:56:13,641 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:56:13,641 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:56:13,641 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:56:13,641 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:56:13,641 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:56:13,641 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:13,641 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. started... 2023-04-19 20:56:13,642 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6873ba93789e63dd9f14391208147267 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-19 20:56:13,642 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6873ba93789e63dd9f14391208147267, store=A 2023-04-19 20:56:13,642 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:56:13,642 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6873ba93789e63dd9f14391208147267, store=B 2023-04-19 20:56:13,642 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:56:13,642 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6873ba93789e63dd9f14391208147267, store=C 2023-04-19 20:56:13,642 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:56:13,655 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041967ecd5fecadf48928395ed882565e32d to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041967ecd5fecadf48928395ed882565e32d 2023-04-19 20:56:13,663 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=199, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/A/adde6acef56446f18b37a68250af7b25 2023-04-19 20:56:13,680 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=199 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/B/3e1978e7f4e945e9add84a619d00c7c2 2023-04-19 20:56:13,702 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=199 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/C/4098d024bfea43c49736438980462d39 2023-04-19 20:56:13,706 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/A/adde6acef56446f18b37a68250af7b25 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/adde6acef56446f18b37a68250af7b25 2023-04-19 20:56:13,709 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/adde6acef56446f18b37a68250af7b25, entries=150, sequenceid=199, filesize=25.3 K 2023-04-19 20:56:13,710 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/B/3e1978e7f4e945e9add84a619d00c7c2 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/3e1978e7f4e945e9add84a619d00c7c2 2023-04-19 20:56:13,715 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/3e1978e7f4e945e9add84a619d00c7c2, entries=150, sequenceid=199, filesize=11.8 K 2023-04-19 20:56:13,715 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/C/4098d024bfea43c49736438980462d39 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/4098d024bfea43c49736438980462d39 2023-04-19 20:56:13,719 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/4098d024bfea43c49736438980462d39, entries=150, sequenceid=199, filesize=11.8 K 2023-04-19 20:56:13,720 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=0 B/0 for 6873ba93789e63dd9f14391208147267 in 78ms, sequenceid=199, compaction requested=true 2023-04-19 20:56:13,720 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6873ba93789e63dd9f14391208147267: 2023-04-19 20:56:13,720 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:13,720 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:56:13,720 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:56:13,720 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:13,720 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:56:13,720 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:56:13,723 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:13,723 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:13,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:13,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:13,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:13,723 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:13,723 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:56:13,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:13,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:13,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:13,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:13,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:13,725 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:56:13,725 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@223cf289[Count = 0] remaining members to acquire global barrier 2023-04-19 20:56:13,725 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:56:13,725 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:13,727 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:13,727 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:13,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:13,727 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:56:13,727 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:56:13,727 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:56:13,728 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:13,728 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:56:13,729 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:56:13,729 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:56:13,729 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:13,729 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:56:13,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:13,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:13,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:13,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:13,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:13,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:13,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:13,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:13,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:13,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:13,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:56:13,732 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:56:13,732 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:56:13,732 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:56:13,732 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:56:13,732 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:56:13,732 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:56:13,734 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:13,734 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:13,734 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:56:13,734 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:13,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:13,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:13,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:13,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:13,734 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:13,734 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:56:13,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:56:13,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:13,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:13,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:13,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:13,736 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:13,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:13,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:13,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:13,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:13,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:13,740 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:13,740 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:56:13,740 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:56:13,740 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:13,740 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:56:13,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:56:13,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:56:13,740 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:56:13,741 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:56:13,741 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:56:13,741 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:56:13,741 DEBUG [Thread-4043] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:56:13,741 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:13,741 DEBUG [Thread-4043] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:56:13,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:56:13,741 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:13,741 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:13,741 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:13,741 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:13,841 DEBUG [Thread-4043] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:56:13,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:56:13,843 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:56:13,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:56:13,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:56:13,845 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:56:13,845 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:56:13,845 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:56:13,845 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:56:13,845 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:13,845 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:13,846 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:13,846 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:56:13,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:56:13,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:56:13,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:13,847 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:13,847 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:56:13,847 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:13,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:56:13,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:13,849 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:56:13,850 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:56:13,850 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:56:13,850 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:56:13,850 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:56:13,850 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:56:13,850 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:56:13,850 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:56:13,850 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:13,850 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. started... 2023-04-19 20:56:13,850 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6873ba93789e63dd9f14391208147267: 2023-04-19 20:56:13,850 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:13,850 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:56:13,850 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:56:13,850 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:13,850 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:56:13,850 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:56:13,852 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:13,852 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:13,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:13,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:13,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:13,853 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:13,853 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:56:13,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:13,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:13,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:13,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:13,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:13,854 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:56:13,854 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ebca216[Count = 0] remaining members to acquire global barrier 2023-04-19 20:56:13,854 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:56:13,854 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:13,855 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:13,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:13,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:13,855 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:56:13,855 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:56:13,855 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:56:13,855 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:13,855 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:56:13,858 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:13,858 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:56:13,858 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:56:13,858 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:56:13,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:13,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:13,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:13,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:13,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:13,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:13,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:13,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:13,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:13,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:13,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:56:13,861 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:56:13,861 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:56:13,861 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:56:13,861 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:56:13,861 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:56:13,861 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:56:13,862 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:13,862 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:13,863 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:13,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:13,863 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:56:13,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:13,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:13,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:13,863 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:56:13,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:56:13,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:13,863 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:13,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:13,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:13,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:13,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:13,864 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:13,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:13,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:13,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:13,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:13,870 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:13,870 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:13,870 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:56:13,870 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:13,870 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:56:13,870 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:13,870 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:56:13,870 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:56:13,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:56:13,870 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:56:13,870 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:56:13,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:56:13,870 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:13,870 DEBUG [Thread-4043] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:56:13,870 DEBUG [Thread-4043] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:56:13,870 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:13,870 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:13,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:56:13,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:56:13,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(9158): Flush requested on 6873ba93789e63dd9f14391208147267 2023-04-19 20:56:13,957 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 6873ba93789e63dd9f14391208147267 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-19 20:56:13,958 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6873ba93789e63dd9f14391208147267, store=A 2023-04-19 20:56:13,958 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:56:13,958 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6873ba93789e63dd9f14391208147267, store=B 2023-04-19 20:56:13,958 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:56:13,958 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6873ba93789e63dd9f14391208147267, store=C 2023-04-19 20:56:13,958 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:56:13,970 DEBUG [Thread-4043] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:56:13,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:56:13,973 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:56:13,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:56:13,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:56:13,974 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:56:13,974 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:56:13,974 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:56:13,974 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:56:13,975 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:13,975 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230419bf3471879a724a238b1cd0c48e33ab0a to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419bf3471879a724a238b1cd0c48e33ab0a 2023-04-19 20:56:13,975 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:13,978 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:56:13,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:56:13,978 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:13,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:56:13,978 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:13,978 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:56:13,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:13,979 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:13,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:56:13,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:13,979 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:56:13,979 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:56:13,979 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:56:13,979 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:56:13,979 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:56:13,979 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:56:13,980 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:56:13,980 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:56:13,980 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:13,980 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. started... 2023-04-19 20:56:13,980 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. as already flushing 2023-04-19 20:56:13,980 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:13,980 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:56:13,980 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:56:13,980 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:13,980 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:56:13,980 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:56:13,981 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:13,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:13,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:13,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:13,982 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:13,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:13,982 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:13,982 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:56:13,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:13,982 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:13,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:13,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57236 deadline: 1681937833976, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:13,983 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:13,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57246 deadline: 1681937833978, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:13,983 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:13,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57226 deadline: 1681937833979, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:13,984 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681937773974, max timestamp=1681937773982Num uniques 859; 2023-04-19 20:56:13,984 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681937773974, max timestamp=1681937773982Num uniques 859; 2023-04-19 20:56:13,984 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-19 20:56:13,985 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681937773974, max timestamp=1681937773982Num uniques 859; 2023-04-19 20:56:13,985 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-19 20:56:13,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:13,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:13,986 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:56:13,986 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@11b9b637[Count = 0] remaining members to acquire global barrier 2023-04-19 20:56:13,984 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-19 20:56:13,986 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:56:13,986 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:13,988 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:13,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:13,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:13,988 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:56:13,988 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:56:13,988 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:56:14,004 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:14,004 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:56:14,006 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:56:14,006 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:14,006 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:56:14,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:14,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:14,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:14,006 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:56:14,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:14,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:14,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:14,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:14,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:14,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:14,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:14,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:56:14,008 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:56:14,008 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:56:14,008 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:56:14,008 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:56:14,008 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:56:14,009 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:56:14,011 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:14,011 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:14,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:14,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:14,012 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:56:14,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:56:14,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:56:14,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:14,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:14,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:14,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:14,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:14,013 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:14,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:14,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:14,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:14,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:14,014 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:14,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:14,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:14,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:14,017 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:14,017 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:56:14,017 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:14,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:56:14,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:56:14,017 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:56:14,017 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:56:14,017 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:56:14,017 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:56:14,017 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:56:14,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:56:14,017 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:14,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:56:14,017 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:14,017 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:14,017 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:14,017 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:14,017 DEBUG [Thread-4043] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:56:14,017 DEBUG [Thread-4043] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:56:14,035 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:14,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57210 deadline: 1681937834031, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:14,035 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:14,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57204 deadline: 1681937834033, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:14,085 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:14,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57236 deadline: 1681937834084, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:14,086 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:14,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57226 deadline: 1681937834085, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:14,090 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:14,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57246 deadline: 1681937834086, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:14,117 DEBUG [Thread-4043] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:56:14,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:56:14,120 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:56:14,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:56:14,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:56:14,121 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:56:14,121 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:56:14,121 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:56:14,121 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:56:14,121 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:14,121 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:14,122 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:14,122 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:56:14,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:56:14,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:56:14,123 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:14,123 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:56:14,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:14,123 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:14,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:56:14,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:14,123 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:56:14,124 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:56:14,124 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:56:14,124 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:56:14,124 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:56:14,124 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:56:14,124 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:56:14,124 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:56:14,124 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:14,124 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. started... 2023-04-19 20:56:14,124 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. as already flushing 2023-04-19 20:56:14,124 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:14,124 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:56:14,124 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:56:14,124 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:14,125 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:56:14,125 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:56:14,126 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:14,126 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:14,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:14,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:14,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:14,126 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:14,126 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:56:14,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:14,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:14,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:14,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:14,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:14,128 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:56:14,128 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1ef7f234[Count = 0] remaining members to acquire global barrier 2023-04-19 20:56:14,128 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:56:14,128 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:14,133 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:14,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:14,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:14,133 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:14,133 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:56:14,133 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:56:14,133 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:56:14,133 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:56:14,134 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:14,134 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:56:14,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:14,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:14,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:14,134 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:56:14,135 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:56:14,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:14,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:14,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:14,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:14,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:14,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:14,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:14,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:56:14,137 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:56:14,137 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:56:14,137 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:56:14,137 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:56:14,137 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:56:14,137 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:56:14,138 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:14,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57210 deadline: 1681937834136, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:14,138 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:14,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57204 deadline: 1681937834136, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:14,139 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:14,139 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:14,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:14,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:14,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:14,139 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:56:14,139 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:14,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:14,139 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:14,139 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:56:14,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:14,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:56:14,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:14,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:14,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:14,141 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:14,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:14,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:14,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:14,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:14,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:14,143 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:56:14,143 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:14,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:56:14,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:56:14,143 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:56:14,143 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:56:14,143 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:14,144 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:14,144 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:14,144 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:14,144 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:14,144 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:14,143 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:56:14,144 DEBUG [Thread-4043] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:56:14,144 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:56:14,143 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:56:14,144 DEBUG [Thread-4043] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:56:14,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:56:14,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:56:14,244 DEBUG [Thread-4043] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:56:14,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:56:14,246 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:56:14,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:56:14,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:56:14,247 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:56:14,247 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:56:14,247 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:56:14,247 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:56:14,248 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:14,248 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:14,250 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:56:14,250 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:14,250 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:56:14,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:56:14,251 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:14,251 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:56:14,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:14,251 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:14,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:56:14,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:14,251 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:56:14,252 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:56:14,252 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:56:14,252 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:56:14,252 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:56:14,252 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:56:14,252 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:56:14,252 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:56:14,252 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:14,252 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. started... 2023-04-19 20:56:14,252 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. as already flushing 2023-04-19 20:56:14,252 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:14,252 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:56:14,253 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:56:14,253 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:14,253 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:56:14,253 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:56:14,254 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:14,254 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:14,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:14,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:14,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:14,255 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:14,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:14,255 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:56:14,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:14,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:14,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:14,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:14,256 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:56:14,256 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5b1ae3d9[Count = 0] remaining members to acquire global barrier 2023-04-19 20:56:14,256 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:56:14,256 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:14,258 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:14,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:14,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:14,258 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:56:14,258 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:56:14,258 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:14,258 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:56:14,258 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:56:14,259 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:14,259 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:56:14,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:14,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:14,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:14,259 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:56:14,259 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:56:14,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:14,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:14,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:14,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:14,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:14,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:14,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:14,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:56:14,262 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:56:14,262 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:56:14,262 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:56:14,262 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:56:14,262 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:56:14,262 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:56:14,264 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:14,264 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:14,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:14,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:14,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:14,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:14,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:14,264 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:56:14,265 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:14,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:56:14,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:56:14,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:14,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:14,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:14,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:14,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:14,266 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:14,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:14,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:14,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:14,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:14,269 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:56:14,269 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:56:14,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:56:14,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:56:14,269 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:14,269 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:56:14,269 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:56:14,269 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:56:14,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:56:14,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:56:14,269 DEBUG [Thread-4043] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:56:14,269 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:14,269 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:14,269 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:14,269 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:14,269 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:14,269 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:14,269 DEBUG [Thread-4043] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:56:14,269 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:56:14,288 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:14,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57226 deadline: 1681937834287, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:14,288 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:14,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57236 deadline: 1681937834287, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:14,293 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:14,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57246 deadline: 1681937834291, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:14,341 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:14,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57210 deadline: 1681937834339, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:14,343 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:14,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57204 deadline: 1681937834339, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:14,370 DEBUG [Thread-4043] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:56:14,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:56:14,372 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:56:14,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:56:14,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:56:14,373 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:56:14,373 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:56:14,373 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:56:14,373 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:56:14,373 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:14,373 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:14,375 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:14,375 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:56:14,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:56:14,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:56:14,375 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:14,375 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:56:14,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:14,376 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:14,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:56:14,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:14,376 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:56:14,376 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:56:14,376 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:56:14,377 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:56:14,377 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:56:14,377 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:56:14,377 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:56:14,377 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:56:14,377 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:14,377 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. started... 2023-04-19 20:56:14,377 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. as already flushing 2023-04-19 20:56:14,377 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:14,377 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:56:14,377 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:56:14,377 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:14,377 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:56:14,377 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:56:14,381 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:14,382 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:14,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:14,382 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=212, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/A/c0c129003c1e49ef96f8542c0e4b0b16 2023-04-19 20:56:14,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:14,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:14,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:14,382 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:14,383 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:56:14,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:14,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:14,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:14,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:14,385 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:56:14,385 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@50bd56be[Count = 0] remaining members to acquire global barrier 2023-04-19 20:56:14,385 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:56:14,385 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:14,386 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:14,386 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:14,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:14,386 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:14,386 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:56:14,386 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:56:14,386 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:56:14,386 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:56:14,389 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:14,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:14,389 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:56:14,389 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:56:14,389 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:56:14,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:14,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:14,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:14,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:14,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:14,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:14,391 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=212 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/B/aa28967d3bcc4f2ab972571953993c0f 2023-04-19 20:56:14,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:14,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:14,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:14,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:56:14,394 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:56:14,394 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:56:14,394 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:56:14,394 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:56:14,394 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:56:14,394 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:56:14,397 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:14,397 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:14,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:14,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:14,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:14,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:14,397 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:56:14,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:14,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:56:14,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:14,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:56:14,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:14,399 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:14,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:14,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:14,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:14,401 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=212 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/C/cc4cecf9f98d402081d04108ed1a6524 2023-04-19 20:56:14,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:14,401 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:14,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:14,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:14,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:14,405 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:56:14,405 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:56:14,405 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/A/c0c129003c1e49ef96f8542c0e4b0b16 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/c0c129003c1e49ef96f8542c0e4b0b16 2023-04-19 20:56:14,405 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:14,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:56:14,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:56:14,405 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:56:14,405 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:56:14,405 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:14,406 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:14,406 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:14,406 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:56:14,405 DEBUG [Thread-4043] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:56:14,406 DEBUG [Thread-4043] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:56:14,405 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:56:14,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:56:14,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:56:14,406 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:14,406 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:14,406 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:14,409 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/c0c129003c1e49ef96f8542c0e4b0b16, entries=450, sequenceid=212, filesize=66.3 K 2023-04-19 20:56:14,409 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/B/aa28967d3bcc4f2ab972571953993c0f as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/aa28967d3bcc4f2ab972571953993c0f 2023-04-19 20:56:14,413 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/aa28967d3bcc4f2ab972571953993c0f, entries=150, sequenceid=212, filesize=11.8 K 2023-04-19 20:56:14,414 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/C/cc4cecf9f98d402081d04108ed1a6524 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/cc4cecf9f98d402081d04108ed1a6524 2023-04-19 20:56:14,417 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/cc4cecf9f98d402081d04108ed1a6524, entries=150, sequenceid=212, filesize=11.8 K 2023-04-19 20:56:14,417 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=120.76 KB/123660 for 6873ba93789e63dd9f14391208147267 in 460ms, sequenceid=212, compaction requested=true 2023-04-19 20:56:14,417 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 6873ba93789e63dd9f14391208147267: 2023-04-19 20:56:14,417 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 20:56:14,418 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-19 20:56:14,418 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-19 20:56:14,418 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 20:56:14,418 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 20:56:14,419 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 145908 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-19 20:56:14,419 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1912): 6873ba93789e63dd9f14391208147267/A is initiating minor compaction (all files) 2023-04-19 20:56:14,419 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 6873ba93789e63dd9f14391208147267/A in TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:14,419 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/350e37412298417db2ae5e7654af6c20, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/ccce7b41508b4527bdf147853f2446dc, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/adde6acef56446f18b37a68250af7b25, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/c0c129003c1e49ef96f8542c0e4b0b16] into tmpdir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp, totalSize=142.5 K 2023-04-19 20:56:14,419 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 350e37412298417db2ae5e7654af6c20, keycount=150, bloomtype=ROW, size=25.7 K, encoding=NONE, compression=NONE, seqNum=147, earliestPutTs=1681937772097 2023-04-19 20:56:14,419 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48520 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-19 20:56:14,419 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1912): 6873ba93789e63dd9f14391208147267/B is initiating minor compaction (all files) 2023-04-19 20:56:14,419 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting ccce7b41508b4527bdf147853f2446dc, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=186, earliestPutTs=1681937772167 2023-04-19 20:56:14,419 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6873ba93789e63dd9f14391208147267/B in TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:14,420 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/eb215aa8e54e4924910d390c3e57fc28, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/12ac88195ff64134a627c06b25f6658c, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/3e1978e7f4e945e9add84a619d00c7c2, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/aa28967d3bcc4f2ab972571953993c0f] into tmpdir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp, totalSize=47.4 K 2023-04-19 20:56:14,420 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting adde6acef56446f18b37a68250af7b25, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=199, earliestPutTs=1681937772792 2023-04-19 20:56:14,420 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting eb215aa8e54e4924910d390c3e57fc28, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=147, earliestPutTs=1681937772097 2023-04-19 20:56:14,420 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting c0c129003c1e49ef96f8542c0e4b0b16, keycount=450, bloomtype=ROW, size=66.3 K, encoding=NONE, compression=NONE, seqNum=212, earliestPutTs=1681937773948 2023-04-19 20:56:14,420 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 12ac88195ff64134a627c06b25f6658c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=186, earliestPutTs=1681937772167 2023-04-19 20:56:14,421 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 3e1978e7f4e945e9add84a619d00c7c2, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=199, earliestPutTs=1681937772792 2023-04-19 20:56:14,421 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting aa28967d3bcc4f2ab972571953993c0f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=212, earliestPutTs=1681937773950 2023-04-19 20:56:14,428 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] throttle.PressureAwareThroughputController(145): 6873ba93789e63dd9f14391208147267#B#compaction#588 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-19 20:56:14,438 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] throttle.PressureAwareThroughputController(145): 6873ba93789e63dd9f14391208147267#A#compaction#587 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 20:56:14,506 DEBUG [Thread-4043] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:56:14,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:56:14,508 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:56:14,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:56:14,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:56:14,509 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:56:14,510 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:56:14,510 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:56:14,510 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:56:14,510 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:14,510 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:14,511 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:56:14,511 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:14,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:56:14,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:56:14,512 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:14,512 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:56:14,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:14,512 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:14,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:56:14,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:14,513 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:56:14,513 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:56:14,513 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:56:14,513 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:56:14,513 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:56:14,513 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:56:14,514 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:56:14,514 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:56:14,514 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:14,514 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. started... 2023-04-19 20:56:14,514 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6873ba93789e63dd9f14391208147267 3/3 column families, dataSize=120.76 KB heapSize=340.45 KB 2023-04-19 20:56:14,514 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6873ba93789e63dd9f14391208147267, store=A 2023-04-19 20:56:14,514 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 20:56:14,515 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6873ba93789e63dd9f14391208147267, store=B 2023-04-19 20:56:14,515 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 20:56:14,515 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6873ba93789e63dd9f14391208147267, store=C 2023-04-19 20:56:14,515 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 20:56:14,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(9158): Flush requested on 6873ba93789e63dd9f14391208147267 2023-04-19 20:56:14,592 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. as already flushing 2023-04-19 20:56:14,606 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:14,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57246 deadline: 1681937834602, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:14,608 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:14,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57226 deadline: 1681937834605, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:14,608 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:14,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57236 deadline: 1681937834605, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:14,645 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:14,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57210 deadline: 1681937834642, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:14,648 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:14,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57204 deadline: 1681937834646, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:14,707 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:14,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57246 deadline: 1681937834707, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:14,711 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:14,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57226 deadline: 1681937834709, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:14,711 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:14,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57236 deadline: 1681937834709, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:14,845 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/B/279d74c15cae4c389475c0e292e93d65 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/279d74c15cae4c389475c0e292e93d65 2023-04-19 20:56:14,849 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 6873ba93789e63dd9f14391208147267/B of 6873ba93789e63dd9f14391208147267 into 279d74c15cae4c389475c0e292e93d65(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 20:56:14,849 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6873ba93789e63dd9f14391208147267: 2023-04-19 20:56:14,849 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267., storeName=6873ba93789e63dd9f14391208147267/B, priority=12, startTime=1681937774418; duration=0sec 2023-04-19 20:56:14,849 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 20:56:14,849 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-19 20:56:14,851 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 60488 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-19 20:56:14,851 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1912): 6873ba93789e63dd9f14391208147267/C is initiating minor compaction (all files) 2023-04-19 20:56:14,851 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6873ba93789e63dd9f14391208147267/C in TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:14,851 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/c55e6ce76d53470f9a91643a0fa72029, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/17dd83e3da5247d9b0da715ea48bda24, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/3fcf93729c9f44ebbb082fda90a8d510, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/4098d024bfea43c49736438980462d39, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/cc4cecf9f98d402081d04108ed1a6524] into tmpdir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp, totalSize=59.1 K 2023-04-19 20:56:14,851 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting c55e6ce76d53470f9a91643a0fa72029, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=137, earliestPutTs=1681937770956 2023-04-19 20:56:14,852 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 17dd83e3da5247d9b0da715ea48bda24, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=147, earliestPutTs=1681937772097 2023-04-19 20:56:14,852 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 3fcf93729c9f44ebbb082fda90a8d510, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=186, earliestPutTs=1681937772167 2023-04-19 20:56:14,852 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 4098d024bfea43c49736438980462d39, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=199, earliestPutTs=1681937772792 2023-04-19 20:56:14,852 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting cc4cecf9f98d402081d04108ed1a6524, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=212, earliestPutTs=1681937773950 2023-04-19 20:56:14,874 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] throttle.PressureAwareThroughputController(145): 6873ba93789e63dd9f14391208147267#C#compaction#590 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 20:56:14,894 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/C/972d94d633ce4e0185cd4ba3093e59ec as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/972d94d633ce4e0185cd4ba3093e59ec 2023-04-19 20:56:14,898 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 6873ba93789e63dd9f14391208147267/C of 6873ba93789e63dd9f14391208147267 into 972d94d633ce4e0185cd4ba3093e59ec(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 20:56:14,898 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6873ba93789e63dd9f14391208147267: 2023-04-19 20:56:14,898 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267., storeName=6873ba93789e63dd9f14391208147267/C, priority=11, startTime=1681937774418; duration=0sec 2023-04-19 20:56:14,899 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 20:56:14,911 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:14,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57246 deadline: 1681937834910, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:14,912 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:14,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57226 deadline: 1681937834912, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:14,916 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:14,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57236 deadline: 1681937834912, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:14,932 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304194ad1ea35507c4845966484b15526bb3e to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304194ad1ea35507c4845966484b15526bb3e 2023-04-19 20:56:14,937 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=246, memsize=40.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/A/d5f423d3d0d54908a0da617f46dee134 2023-04-19 20:56:14,946 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=246 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/B/0988ed0ae02249b6a440b2ae5400dea1 2023-04-19 20:56:14,955 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=246 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/C/410bffe0cea7475383bc553c8a8923ed 2023-04-19 20:56:14,959 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/A/d5f423d3d0d54908a0da617f46dee134 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/d5f423d3d0d54908a0da617f46dee134 2023-04-19 20:56:14,961 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/d5f423d3d0d54908a0da617f46dee134, entries=150, sequenceid=246, filesize=25.3 K 2023-04-19 20:56:14,962 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/B/0988ed0ae02249b6a440b2ae5400dea1 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/0988ed0ae02249b6a440b2ae5400dea1 2023-04-19 20:56:14,965 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/0988ed0ae02249b6a440b2ae5400dea1, entries=150, sequenceid=246, filesize=11.8 K 2023-04-19 20:56:14,966 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/C/410bffe0cea7475383bc553c8a8923ed as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/410bffe0cea7475383bc553c8a8923ed 2023-04-19 20:56:14,969 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/410bffe0cea7475383bc553c8a8923ed, entries=150, sequenceid=246, filesize=11.8 K 2023-04-19 20:56:14,970 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~340.41 KB/348576, currentSize=60.38 KB/61830 for 6873ba93789e63dd9f14391208147267 in 456ms, sequenceid=246, compaction requested=false 2023-04-19 20:56:14,970 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6873ba93789e63dd9f14391208147267: 2023-04-19 20:56:14,970 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:14,970 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:56:14,970 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:56:14,970 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:14,970 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:56:14,970 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:56:14,972 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:14,972 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:14,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:14,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:14,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:14,972 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:14,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:14,972 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:56:14,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:14,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:14,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:14,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:14,975 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:56:14,975 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@c695a90[Count = 0] remaining members to acquire global barrier 2023-04-19 20:56:14,975 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:56:14,975 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:14,976 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:14,976 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:14,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:14,976 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:56:14,976 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:56:14,976 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:56:14,976 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:14,976 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:56:14,978 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:56:14,978 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:56:14,978 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:56:14,978 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:14,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:14,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:14,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:14,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:14,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:14,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:14,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:14,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:14,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:14,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:14,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:56:14,981 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:56:14,981 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:56:14,981 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:56:14,981 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:56:14,981 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:56:14,981 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:56:14,983 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:14,983 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:14,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:14,983 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:14,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:14,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:14,983 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:56:14,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:14,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:14,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:56:14,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:14,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:56:14,984 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:14,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:14,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:14,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:14,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:14,986 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:14,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:14,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:14,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:14,988 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:56:14,988 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:56:14,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:56:14,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:56:14,988 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:14,988 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:14,988 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:14,988 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:14,988 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:14,988 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:56:14,988 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:56:14,988 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:14,988 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:56:14,988 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:56:14,988 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:14,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:56:14,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:56:14,989 DEBUG [Thread-4043] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:56:14,989 DEBUG [Thread-4043] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:56:15,089 DEBUG [Thread-4043] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:56:15,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:56:15,091 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:56:15,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:56:15,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:56:15,092 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:56:15,092 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:56:15,092 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:56:15,092 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:56:15,093 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:15,093 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:15,094 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:56:15,094 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:15,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:56:15,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:56:15,094 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:15,094 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:56:15,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:15,095 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:15,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:56:15,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:15,095 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:56:15,095 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:56:15,096 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:56:15,096 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:56:15,096 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:56:15,096 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:56:15,096 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:56:15,096 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:56:15,096 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:15,096 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. started... 2023-04-19 20:56:15,096 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6873ba93789e63dd9f14391208147267 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-19 20:56:15,096 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6873ba93789e63dd9f14391208147267, store=A 2023-04-19 20:56:15,096 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:56:15,097 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6873ba93789e63dd9f14391208147267, store=B 2023-04-19 20:56:15,097 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:56:15,097 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6873ba93789e63dd9f14391208147267, store=C 2023-04-19 20:56:15,097 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:56:15,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(9158): Flush requested on 6873ba93789e63dd9f14391208147267 2023-04-19 20:56:15,148 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. as already flushing 2023-04-19 20:56:15,179 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:15,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57210 deadline: 1681937835177, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:15,181 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:15,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57204 deadline: 1681937835179, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:15,215 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:15,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57226 deadline: 1681937835214, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:15,215 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:15,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57246 deadline: 1681937835215, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:15,220 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:15,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57236 deadline: 1681937835219, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:15,282 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:15,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57210 deadline: 1681937835280, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:15,283 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:15,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57204 deadline: 1681937835282, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:15,484 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:15,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57210 deadline: 1681937835483, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:15,486 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:15,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57204 deadline: 1681937835484, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:15,513 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304191bb16a345dfe4f1c8c83892e941a7f99 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304191bb16a345dfe4f1c8c83892e941a7f99 2023-04-19 20:56:15,691 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/A/4c632f2df9dc4efbb2aecf0c495aa7f5 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/4c632f2df9dc4efbb2aecf0c495aa7f5 2023-04-19 20:56:15,695 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 6873ba93789e63dd9f14391208147267/A of 6873ba93789e63dd9f14391208147267 into 4c632f2df9dc4efbb2aecf0c495aa7f5(size=25.8 K), total size for store is 51.1 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-19 20:56:15,695 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6873ba93789e63dd9f14391208147267: 2023-04-19 20:56:15,695 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267., storeName=6873ba93789e63dd9f14391208147267/A, priority=12, startTime=1681937774417; duration=1sec 2023-04-19 20:56:15,695 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 20:56:15,720 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:15,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57246 deadline: 1681937835717, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:15,722 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:15,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57226 deadline: 1681937835720, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:15,727 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:15,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57236 deadline: 1681937835725, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:15,787 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:15,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57210 deadline: 1681937835785, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:15,791 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:15,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57204 deadline: 1681937835788, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:15,918 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=260, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/A/8e1a60905ca2404d90b6b0533e03aaa0 2023-04-19 20:56:16,293 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:16,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57210 deadline: 1681937836291, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:16,297 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:16,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57204 deadline: 1681937836296, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:16,328 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=260 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/B/5278cdc7792540ae9f91bb8cbbc8a1c6 2023-04-19 20:56:16,336 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=260 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/C/a3fd7c0f1b884fa79ddb4b4169951bc5 2023-04-19 20:56:16,340 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/A/8e1a60905ca2404d90b6b0533e03aaa0 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/8e1a60905ca2404d90b6b0533e03aaa0 2023-04-19 20:56:16,343 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/8e1a60905ca2404d90b6b0533e03aaa0, entries=150, sequenceid=260, filesize=25.3 K 2023-04-19 20:56:16,344 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/B/5278cdc7792540ae9f91bb8cbbc8a1c6 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/5278cdc7792540ae9f91bb8cbbc8a1c6 2023-04-19 20:56:16,346 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/5278cdc7792540ae9f91bb8cbbc8a1c6, entries=150, sequenceid=260, filesize=11.8 K 2023-04-19 20:56:16,347 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/C/a3fd7c0f1b884fa79ddb4b4169951bc5 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/a3fd7c0f1b884fa79ddb4b4169951bc5 2023-04-19 20:56:16,350 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/a3fd7c0f1b884fa79ddb4b4169951bc5, entries=150, sequenceid=260, filesize=11.8 K 2023-04-19 20:56:16,351 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 6873ba93789e63dd9f14391208147267 in 1255ms, sequenceid=260, compaction requested=true 2023-04-19 20:56:16,351 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6873ba93789e63dd9f14391208147267: 2023-04-19 20:56:16,351 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:16,351 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:56:16,351 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:56:16,351 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:16,351 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:56:16,351 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:56:16,354 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:16,354 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:16,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:16,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:16,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:16,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:16,354 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:16,354 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:56:16,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:16,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:16,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:16,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:16,355 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:56:16,355 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2dae7f7b[Count = 0] remaining members to acquire global barrier 2023-04-19 20:56:16,355 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:56:16,355 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:16,357 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:16,357 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:16,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:16,357 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:56:16,357 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:56:16,357 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:56:16,358 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:16,358 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:56:16,359 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:16,359 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:56:16,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:16,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:16,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:16,359 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:56:16,359 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:56:16,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:16,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:16,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:16,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:16,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:16,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:16,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:16,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:56:16,362 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:56:16,362 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:56:16,362 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:56:16,362 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:56:16,362 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:56:16,362 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:56:16,363 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:16,363 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:56:16,363 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:16,363 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:16,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:16,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:16,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:16,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:16,364 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:16,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:16,364 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:56:16,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:56:16,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:16,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:16,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:16,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:16,365 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:16,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:16,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:16,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:16,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:16,368 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:56:16,368 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:56:16,368 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:56:16,368 DEBUG [Thread-4043] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:56:16,368 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:16,368 DEBUG [Thread-4043] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:56:16,368 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:56:16,368 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:56:16,368 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:16,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:56:16,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:56:16,369 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:16,369 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:16,369 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:16,369 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:16,369 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:16,368 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:56:16,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:56:16,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:56:16,468 DEBUG [Thread-4043] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:56:16,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:56:16,470 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:56:16,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:56:16,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:56:16,472 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:56:16,472 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:56:16,472 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:56:16,472 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:56:16,472 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:16,472 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:16,474 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:56:16,474 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:16,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:56:16,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:56:16,475 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:16,475 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:56:16,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:16,475 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:16,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:56:16,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:16,476 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:56:16,476 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:56:16,476 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:56:16,476 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:56:16,476 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:56:16,476 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:56:16,476 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:56:16,476 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:56:16,476 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:16,476 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. started... 2023-04-19 20:56:16,477 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6873ba93789e63dd9f14391208147267 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-19 20:56:16,477 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6873ba93789e63dd9f14391208147267, store=A 2023-04-19 20:56:16,477 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:56:16,477 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6873ba93789e63dd9f14391208147267, store=B 2023-04-19 20:56:16,477 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:56:16,477 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6873ba93789e63dd9f14391208147267, store=C 2023-04-19 20:56:16,477 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:56:16,489 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304195ec0d5c5608d43a7a9c3e5cbbf3e5335 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304195ec0d5c5608d43a7a9c3e5cbbf3e5335 2023-04-19 20:56:16,495 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=281, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/A/113fc71247004215b1dca5f5e79eec30 2023-04-19 20:56:16,504 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=281 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/B/cf6e538d539e4db8b45d1e38aef27620 2023-04-19 20:56:16,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(9158): Flush requested on 6873ba93789e63dd9f14391208147267 2023-04-19 20:56:16,725 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. as already flushing 2023-04-19 20:56:16,739 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:16,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57246 deadline: 1681937836737, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:16,741 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:16,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57226 deadline: 1681937836738, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:16,741 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:16,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57236 deadline: 1681937836739, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:16,842 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:16,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57246 deadline: 1681937836840, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:16,845 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:16,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57236 deadline: 1681937836843, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:16,845 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:16,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57226 deadline: 1681937836844, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:16,918 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=281 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/C/0b52a0554f824d179867e0bcf5ff4ecf 2023-04-19 20:56:16,924 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/A/113fc71247004215b1dca5f5e79eec30 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/113fc71247004215b1dca5f5e79eec30 2023-04-19 20:56:16,927 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/113fc71247004215b1dca5f5e79eec30, entries=150, sequenceid=281, filesize=25.4 K 2023-04-19 20:56:16,928 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/B/cf6e538d539e4db8b45d1e38aef27620 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/cf6e538d539e4db8b45d1e38aef27620 2023-04-19 20:56:16,931 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/cf6e538d539e4db8b45d1e38aef27620, entries=150, sequenceid=281, filesize=11.9 K 2023-04-19 20:56:16,932 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/C/0b52a0554f824d179867e0bcf5ff4ecf as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/0b52a0554f824d179867e0bcf5ff4ecf 2023-04-19 20:56:16,935 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/0b52a0554f824d179867e0bcf5ff4ecf, entries=150, sequenceid=281, filesize=11.9 K 2023-04-19 20:56:16,936 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 6873ba93789e63dd9f14391208147267 in 460ms, sequenceid=281, compaction requested=true 2023-04-19 20:56:16,936 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6873ba93789e63dd9f14391208147267: 2023-04-19 20:56:16,936 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:16,936 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:56:16,936 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:56:16,936 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:16,936 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:56:16,936 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:56:16,938 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:16,938 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:16,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:16,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:16,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:16,938 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:16,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:16,939 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:56:16,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:16,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:16,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:16,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:16,940 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:56:16,940 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2126553c[Count = 0] remaining members to acquire global barrier 2023-04-19 20:56:16,940 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:56:16,940 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:16,944 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:16,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:16,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:16,944 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:56:16,944 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:56:16,944 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:56:16,945 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:16,945 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:56:16,947 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:16,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:16,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:16,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:16,947 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:56:16,947 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:56:16,947 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:56:16,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:16,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:16,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:16,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:16,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:16,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:16,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:16,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:56:16,950 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:56:16,950 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:56:16,950 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:56:16,950 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:56:16,950 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:56:16,950 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:56:16,951 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:16,952 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:16,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:16,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:16,952 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:56:16,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:16,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:16,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:16,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:56:16,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:56:16,952 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:16,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:16,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:16,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:16,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:16,954 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:16,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:16,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:16,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:16,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:16,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:16,957 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:16,957 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:16,957 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:16,957 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:16,957 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:56:16,958 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:56:16,957 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:56:16,958 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:56:16,957 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:16,958 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:16,958 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:16,957 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:56:16,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:56:16,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:56:16,958 DEBUG [Thread-4043] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:56:16,958 DEBUG [Thread-4043] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:56:16,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:56:16,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:56:17,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(9158): Flush requested on 6873ba93789e63dd9f14391208147267 2023-04-19 20:56:17,046 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 6873ba93789e63dd9f14391208147267 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-19 20:56:17,047 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6873ba93789e63dd9f14391208147267, store=A 2023-04-19 20:56:17,047 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:56:17,047 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6873ba93789e63dd9f14391208147267, store=B 2023-04-19 20:56:17,047 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:56:17,047 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6873ba93789e63dd9f14391208147267, store=C 2023-04-19 20:56:17,047 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:56:17,058 DEBUG [Thread-4043] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:56:17,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:56:17,059 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304192fff06512e764eeb8e6e2e2f6c39e2a0 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304192fff06512e764eeb8e6e2e2f6c39e2a0 2023-04-19 20:56:17,060 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:56:17,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:56:17,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:56:17,062 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:56:17,062 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:56:17,062 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:56:17,062 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:56:17,063 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:17,063 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:17,064 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:56:17,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:56:17,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:56:17,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:17,065 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:17,065 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:17,065 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:17,065 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:56:17,065 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=294, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/A/301e9045dce443a7bfaa2b07a2816403 2023-04-19 20:56:17,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:56:17,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:17,066 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:56:17,066 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:56:17,066 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:56:17,066 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:56:17,066 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:56:17,066 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:56:17,066 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:17,066 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. started... 2023-04-19 20:56:17,066 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:56:17,066 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. as already flushing 2023-04-19 20:56:17,067 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:17,067 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:56:17,067 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:56:17,067 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:17,067 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:56:17,067 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:56:17,069 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:17,069 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:17,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:17,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:17,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:17,069 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:17,069 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:56:17,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:17,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:17,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:17,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:17,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:17,072 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:56:17,072 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1b94ef00[Count = 0] remaining members to acquire global barrier 2023-04-19 20:56:17,072 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:56:17,072 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:17,073 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:17,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:17,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:17,073 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:56:17,073 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:56:17,073 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:56:17,073 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:17,073 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:56:17,074 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:17,075 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:56:17,075 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:56:17,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:17,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:17,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:17,075 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:56:17,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:17,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:17,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:17,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:17,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:17,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:17,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:17,077 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:17,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57246 deadline: 1681937837073, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:17,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:56:17,078 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:56:17,078 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:56:17,078 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:56:17,078 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:56:17,078 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:56:17,078 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:56:17,079 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:17,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57226 deadline: 1681937837076, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:17,080 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:17,080 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:17,080 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:56:17,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:17,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:17,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:17,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:17,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:17,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:17,081 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:56:17,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:56:17,081 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:17,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:17,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:17,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:17,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:17,082 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:17,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:17,082 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:17,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:17,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57236 deadline: 1681937837078, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:17,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:17,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:17,085 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:17,085 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:17,085 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:17,085 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:17,085 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:17,085 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:17,085 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:56:17,086 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:56:17,086 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:56:17,086 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:56:17,085 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:56:17,085 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:17,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:56:17,086 DEBUG [Thread-4043] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:56:17,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:56:17,086 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:56:17,086 DEBUG [Thread-4043] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:56:17,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:56:17,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:56:17,181 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:17,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57246 deadline: 1681937837179, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:17,182 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:17,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57226 deadline: 1681937837180, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:17,185 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:17,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57236 deadline: 1681937837183, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:17,186 DEBUG [Thread-4043] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:56:17,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:56:17,188 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:56:17,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:56:17,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:56:17,190 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:56:17,190 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:56:17,190 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:56:17,190 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:56:17,190 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:17,190 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:17,192 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:17,192 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:56:17,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:56:17,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:56:17,192 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:17,192 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:56:17,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:17,192 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:17,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:56:17,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:17,193 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:56:17,193 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:56:17,193 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:56:17,193 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:56:17,193 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:56:17,193 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:56:17,194 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:56:17,194 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:56:17,194 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:17,194 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. started... 2023-04-19 20:56:17,194 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. as already flushing 2023-04-19 20:56:17,194 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:17,194 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:56:17,194 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:56:17,194 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:17,194 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:56:17,194 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:56:17,196 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:17,196 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:17,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:17,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:17,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:17,196 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:17,196 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:56:17,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:17,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:17,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:17,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:17,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:17,198 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:56:17,198 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5def8b38[Count = 0] remaining members to acquire global barrier 2023-04-19 20:56:17,198 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:56:17,198 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:17,199 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:17,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:17,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:17,200 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:56:17,200 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:56:17,200 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:56:17,200 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:17,200 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:56:17,202 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:56:17,202 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:56:17,202 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:17,202 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:56:17,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:17,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:17,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:17,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:17,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:17,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:17,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:17,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:17,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:17,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:17,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:56:17,205 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:56:17,205 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:56:17,205 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:56:17,205 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:56:17,205 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:56:17,205 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:56:17,207 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:17,207 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:56:17,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:17,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:17,207 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:17,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:17,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:17,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:17,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:56:17,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:56:17,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:17,208 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:17,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:17,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:17,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:17,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:17,209 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:17,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:17,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:17,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:17,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:17,211 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:56:17,212 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:56:17,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:56:17,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:56:17,212 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:17,212 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:17,212 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:17,212 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:56:17,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:56:17,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:56:17,212 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:56:17,212 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:56:17,212 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:56:17,212 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:17,212 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:17,212 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:17,212 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:17,213 DEBUG [Thread-4043] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:56:17,213 DEBUG [Thread-4043] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:56:17,297 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:17,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57210 deadline: 1681937837295, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:17,302 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:17,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57204 deadline: 1681937837301, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:17,313 DEBUG [Thread-4043] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:56:17,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:56:17,315 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:56:17,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:56:17,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:56:17,317 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:56:17,317 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:56:17,317 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:56:17,317 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:56:17,318 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:17,318 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:17,319 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:17,319 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:56:17,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:56:17,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:56:17,319 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:17,319 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:56:17,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:17,320 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:17,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:56:17,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:17,320 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:56:17,320 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:56:17,320 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:56:17,320 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:56:17,320 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:56:17,321 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:56:17,321 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:56:17,321 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:56:17,321 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:17,321 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. started... 2023-04-19 20:56:17,321 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. as already flushing 2023-04-19 20:56:17,321 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:17,321 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:56:17,321 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:56:17,321 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:17,321 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:56:17,321 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:56:17,324 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:17,324 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:17,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:17,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:17,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:17,324 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:17,324 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:56:17,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:17,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:17,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:17,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:17,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:17,326 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:56:17,326 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3e46259[Count = 0] remaining members to acquire global barrier 2023-04-19 20:56:17,326 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:56:17,326 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:17,328 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:17,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:17,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:17,328 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:56:17,328 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:56:17,328 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:17,328 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:56:17,328 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:56:17,332 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:56:17,332 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:56:17,332 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:17,333 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:56:17,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:17,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:17,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:17,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:17,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:17,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:17,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:17,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:17,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:17,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:17,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:56:17,336 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:56:17,336 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:56:17,336 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:56:17,336 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:56:17,336 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:56:17,336 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:56:17,344 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:17,344 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:17,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:17,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:17,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:17,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:17,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:17,344 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:56:17,344 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:17,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:56:17,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:56:17,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:17,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:17,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:17,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:17,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:17,346 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:17,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:17,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:17,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:17,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:17,349 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:56:17,349 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:56:17,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:56:17,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:56:17,349 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:17,349 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:17,349 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:17,349 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:56:17,349 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:56:17,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:56:17,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:56:17,349 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:56:17,349 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:56:17,349 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:17,349 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:17,349 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:17,349 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:17,350 DEBUG [Thread-4043] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:56:17,350 DEBUG [Thread-4043] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:56:17,383 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:17,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57246 deadline: 1681937837382, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:17,386 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:17,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57226 deadline: 1681937837384, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:17,389 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:17,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57236 deadline: 1681937837387, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:17,450 DEBUG [Thread-4043] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:56:17,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:56:17,452 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:56:17,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:56:17,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:56:17,453 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:56:17,453 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:56:17,453 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:56:17,453 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:56:17,454 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:17,454 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:17,455 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:17,455 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:56:17,455 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:56:17,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:56:17,455 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:17,455 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:56:17,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:17,456 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:17,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:56:17,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:17,456 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:56:17,456 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:56:17,456 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:56:17,457 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:56:17,457 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:56:17,457 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:56:17,457 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:56:17,457 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:56:17,457 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:17,457 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. started... 2023-04-19 20:56:17,457 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. as already flushing 2023-04-19 20:56:17,457 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:17,457 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:56:17,458 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:56:17,458 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:17,458 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:56:17,458 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:56:17,459 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:17,459 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:17,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:17,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:17,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:17,459 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:17,459 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:56:17,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:17,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:17,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:17,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:17,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:17,461 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:56:17,461 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@10f848ac[Count = 0] remaining members to acquire global barrier 2023-04-19 20:56:17,461 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:56:17,461 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:17,463 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:17,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:17,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:17,463 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:56:17,463 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:56:17,463 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:56:17,463 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:17,463 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:56:17,464 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:17,464 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:56:17,465 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:56:17,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:17,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:17,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:17,465 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:56:17,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:17,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:17,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:17,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:17,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:17,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:17,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:17,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:56:17,467 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:56:17,467 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:56:17,467 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:56:17,467 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:56:17,467 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:56:17,467 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:56:17,468 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:17,468 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:56:17,468 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:17,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:17,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:17,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:17,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:17,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:17,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:17,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:56:17,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:56:17,469 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:17,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:17,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:17,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:17,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:17,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:17,470 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:17,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:17,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:17,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:17,472 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:17,472 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:17,472 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:17,472 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:17,472 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:17,472 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:56:17,472 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:17,473 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:17,473 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:56:17,473 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:56:17,473 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:56:17,473 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:56:17,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:56:17,473 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:56:17,473 DEBUG [Thread-4043] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:56:17,473 DEBUG [Thread-4043] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:56:17,473 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:56:17,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:56:17,474 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:56:17,477 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=294 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/B/25eec6e070474d5fb14e4fcd26f4b4c3 2023-04-19 20:56:17,488 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=294 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/C/baa0548f46a846b4b48e1a25e7f3913b 2023-04-19 20:56:17,492 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/A/301e9045dce443a7bfaa2b07a2816403 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/301e9045dce443a7bfaa2b07a2816403 2023-04-19 20:56:17,496 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/301e9045dce443a7bfaa2b07a2816403, entries=150, sequenceid=294, filesize=25.4 K 2023-04-19 20:56:17,497 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/B/25eec6e070474d5fb14e4fcd26f4b4c3 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/25eec6e070474d5fb14e4fcd26f4b4c3 2023-04-19 20:56:17,501 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/25eec6e070474d5fb14e4fcd26f4b4c3, entries=150, sequenceid=294, filesize=11.9 K 2023-04-19 20:56:17,502 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/C/baa0548f46a846b4b48e1a25e7f3913b as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/baa0548f46a846b4b48e1a25e7f3913b 2023-04-19 20:56:17,505 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/baa0548f46a846b4b48e1a25e7f3913b, entries=150, sequenceid=294, filesize=11.9 K 2023-04-19 20:56:17,505 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 6873ba93789e63dd9f14391208147267 in 459ms, sequenceid=294, compaction requested=true 2023-04-19 20:56:17,505 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 6873ba93789e63dd9f14391208147267: 2023-04-19 20:56:17,505 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 20:56:17,505 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 20:56:17,505 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 20:56:17,505 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-19 20:56:17,505 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-19 20:56:17,507 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 130342 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-19 20:56:17,507 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61042 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-19 20:56:17,507 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1912): 6873ba93789e63dd9f14391208147267/A is initiating minor compaction (all files) 2023-04-19 20:56:17,507 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1912): 6873ba93789e63dd9f14391208147267/B is initiating minor compaction (all files) 2023-04-19 20:56:17,507 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 6873ba93789e63dd9f14391208147267/A in TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:17,507 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6873ba93789e63dd9f14391208147267/B in TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:17,507 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/4c632f2df9dc4efbb2aecf0c495aa7f5, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/d5f423d3d0d54908a0da617f46dee134, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/8e1a60905ca2404d90b6b0533e03aaa0, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/113fc71247004215b1dca5f5e79eec30, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/301e9045dce443a7bfaa2b07a2816403] into tmpdir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp, totalSize=127.3 K 2023-04-19 20:56:17,507 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/279d74c15cae4c389475c0e292e93d65, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/0988ed0ae02249b6a440b2ae5400dea1, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/5278cdc7792540ae9f91bb8cbbc8a1c6, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/cf6e538d539e4db8b45d1e38aef27620, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/25eec6e070474d5fb14e4fcd26f4b4c3] into tmpdir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp, totalSize=59.6 K 2023-04-19 20:56:17,508 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 4c632f2df9dc4efbb2aecf0c495aa7f5, keycount=150, bloomtype=ROW, size=25.8 K, encoding=NONE, compression=NONE, seqNum=212, earliestPutTs=1681937773950 2023-04-19 20:56:17,508 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 279d74c15cae4c389475c0e292e93d65, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=212, earliestPutTs=1681937773950 2023-04-19 20:56:17,508 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting d5f423d3d0d54908a0da617f46dee134, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=246, earliestPutTs=1681937773987 2023-04-19 20:56:17,508 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 0988ed0ae02249b6a440b2ae5400dea1, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=246, earliestPutTs=1681937773987 2023-04-19 20:56:17,508 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 8e1a60905ca2404d90b6b0533e03aaa0, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=260, earliestPutTs=1681937774595 2023-04-19 20:56:17,509 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 5278cdc7792540ae9f91bb8cbbc8a1c6, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=260, earliestPutTs=1681937774595 2023-04-19 20:56:17,509 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 113fc71247004215b1dca5f5e79eec30, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=281, earliestPutTs=1681937775175 2023-04-19 20:56:17,509 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting cf6e538d539e4db8b45d1e38aef27620, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=281, earliestPutTs=1681937775175 2023-04-19 20:56:17,509 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 301e9045dce443a7bfaa2b07a2816403, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=294, earliestPutTs=1681937776732 2023-04-19 20:56:17,510 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 25eec6e070474d5fb14e4fcd26f4b4c3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=294, earliestPutTs=1681937776732 2023-04-19 20:56:17,520 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] throttle.PressureAwareThroughputController(145): 6873ba93789e63dd9f14391208147267#B#compaction#602 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-19 20:56:17,524 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] throttle.PressureAwareThroughputController(145): 6873ba93789e63dd9f14391208147267#A#compaction#603 average throughput is 4.93 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 20:56:17,542 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/B/6b7e2a62e336464d9940c09feb96e14f as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/6b7e2a62e336464d9940c09feb96e14f 2023-04-19 20:56:17,548 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 6873ba93789e63dd9f14391208147267/B of 6873ba93789e63dd9f14391208147267 into 6b7e2a62e336464d9940c09feb96e14f(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 20:56:17,548 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6873ba93789e63dd9f14391208147267: 2023-04-19 20:56:17,549 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267., storeName=6873ba93789e63dd9f14391208147267/B, priority=11, startTime=1681937777505; duration=0sec 2023-04-19 20:56:17,549 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 20:56:17,549 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-19 20:56:17,550 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61008 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-19 20:56:17,551 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1912): 6873ba93789e63dd9f14391208147267/C is initiating minor compaction (all files) 2023-04-19 20:56:17,551 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6873ba93789e63dd9f14391208147267/C in TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:17,551 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/972d94d633ce4e0185cd4ba3093e59ec, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/410bffe0cea7475383bc553c8a8923ed, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/a3fd7c0f1b884fa79ddb4b4169951bc5, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/0b52a0554f824d179867e0bcf5ff4ecf, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/baa0548f46a846b4b48e1a25e7f3913b] into tmpdir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp, totalSize=59.6 K 2023-04-19 20:56:17,551 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 972d94d633ce4e0185cd4ba3093e59ec, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=212, earliestPutTs=1681937773950 2023-04-19 20:56:17,551 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 410bffe0cea7475383bc553c8a8923ed, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=246, earliestPutTs=1681937773987 2023-04-19 20:56:17,551 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting a3fd7c0f1b884fa79ddb4b4169951bc5, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=260, earliestPutTs=1681937774595 2023-04-19 20:56:17,551 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 0b52a0554f824d179867e0bcf5ff4ecf, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=281, earliestPutTs=1681937775175 2023-04-19 20:56:17,552 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting baa0548f46a846b4b48e1a25e7f3913b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=294, earliestPutTs=1681937776732 2023-04-19 20:56:17,569 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] throttle.PressureAwareThroughputController(145): 6873ba93789e63dd9f14391208147267#C#compaction#604 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 20:56:17,573 DEBUG [Thread-4043] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:56:17,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:56:17,575 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:56:17,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:56:17,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:56:17,584 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:56:17,584 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:56:17,584 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:56:17,584 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:56:17,585 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:17,585 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:17,587 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:56:17,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:56:17,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:56:17,587 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:17,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:17,588 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:17,588 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:56:17,589 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:17,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:56:17,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:17,589 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:56:17,589 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:56:17,589 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:56:17,590 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:56:17,590 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:56:17,590 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:56:17,594 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/C/caf2f2e21ba949b2bbd08822eff3eb91 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/caf2f2e21ba949b2bbd08822eff3eb91 2023-04-19 20:56:17,598 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 6873ba93789e63dd9f14391208147267/C of 6873ba93789e63dd9f14391208147267 into caf2f2e21ba949b2bbd08822eff3eb91(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 20:56:17,598 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6873ba93789e63dd9f14391208147267: 2023-04-19 20:56:17,598 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267., storeName=6873ba93789e63dd9f14391208147267/C, priority=11, startTime=1681937777505; duration=0sec 2023-04-19 20:56:17,598 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 20:56:17,600 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:56:17,600 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:56:17,600 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:17,600 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. started... 2023-04-19 20:56:17,600 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6873ba93789e63dd9f14391208147267 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-19 20:56:17,600 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6873ba93789e63dd9f14391208147267, store=A 2023-04-19 20:56:17,600 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:56:17,600 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6873ba93789e63dd9f14391208147267, store=B 2023-04-19 20:56:17,600 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:56:17,600 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6873ba93789e63dd9f14391208147267, store=C 2023-04-19 20:56:17,600 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:56:17,624 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304195ae8e0bd317f436e8dae5510adb191a4 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304195ae8e0bd317f436e8dae5510adb191a4 2023-04-19 20:56:17,633 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=315, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/A/607f4960866b4b768437363b25dd3849 2023-04-19 20:56:17,647 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=315 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/B/9750333ee29047d2af6c880445330aeb 2023-04-19 20:56:17,658 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=315 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/C/df3c8da2747646e589d8c282d19e756b 2023-04-19 20:56:17,663 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/A/607f4960866b4b768437363b25dd3849 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/607f4960866b4b768437363b25dd3849 2023-04-19 20:56:17,667 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/607f4960866b4b768437363b25dd3849, entries=150, sequenceid=315, filesize=25.4 K 2023-04-19 20:56:17,668 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/B/9750333ee29047d2af6c880445330aeb as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/9750333ee29047d2af6c880445330aeb 2023-04-19 20:56:17,671 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/9750333ee29047d2af6c880445330aeb, entries=150, sequenceid=315, filesize=11.9 K 2023-04-19 20:56:17,672 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/C/df3c8da2747646e589d8c282d19e756b as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/df3c8da2747646e589d8c282d19e756b 2023-04-19 20:56:17,675 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/df3c8da2747646e589d8c282d19e756b, entries=150, sequenceid=315, filesize=11.9 K 2023-04-19 20:56:17,676 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=0 B/0 for 6873ba93789e63dd9f14391208147267 in 76ms, sequenceid=315, compaction requested=false 2023-04-19 20:56:17,676 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6873ba93789e63dd9f14391208147267: 2023-04-19 20:56:17,676 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:17,676 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:56:17,676 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:56:17,676 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:17,676 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:56:17,676 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:56:17,678 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:17,678 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:17,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:17,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:17,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:17,678 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:17,678 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:56:17,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:17,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:17,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:17,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:17,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:17,680 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:56:17,680 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2ca8495c[Count = 0] remaining members to acquire global barrier 2023-04-19 20:56:17,680 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:56:17,680 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:17,682 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:17,682 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:17,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:17,682 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:56:17,682 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:56:17,682 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:56:17,682 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:17,682 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:56:17,683 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:56:17,683 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:56:17,683 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:56:17,684 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:17,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:17,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:17,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:17,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:17,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:17,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:17,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:17,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:17,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:17,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:17,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:56:17,686 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:56:17,687 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:56:17,687 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:56:17,687 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:56:17,687 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:56:17,687 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:56:17,689 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:17,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:17,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:17,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:17,689 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:17,690 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:56:17,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:17,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:17,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:17,690 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:17,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:56:17,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:56:17,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:17,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:17,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:17,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:17,691 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:17,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:17,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:17,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:17,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:17,694 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:17,694 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:17,694 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:56:17,694 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:56:17,694 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:56:17,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:56:17,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:56:17,694 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:56:17,694 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:56:17,694 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:56:17,694 DEBUG [Thread-4043] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:56:17,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:56:17,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:56:17,694 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:17,695 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:17,695 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:17,695 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:17,695 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:17,695 DEBUG [Thread-4043] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:56:17,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(9158): Flush requested on 6873ba93789e63dd9f14391208147267 2023-04-19 20:56:17,723 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 6873ba93789e63dd9f14391208147267 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-19 20:56:17,723 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6873ba93789e63dd9f14391208147267, store=A 2023-04-19 20:56:17,723 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:56:17,723 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6873ba93789e63dd9f14391208147267, store=B 2023-04-19 20:56:17,723 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:56:17,723 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6873ba93789e63dd9f14391208147267, store=C 2023-04-19 20:56:17,723 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:56:17,773 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681937777759, max timestamp=1681937777771Num uniques 859; 2023-04-19 20:56:17,773 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 20:56:17,773 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681937777759, max timestamp=1681937777771Num uniques 859; 2023-04-19 20:56:17,773 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681937777759, max timestamp=1681937777771Num uniques 859; 2023-04-19 20:56:17,773 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 20:56:17,773 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 20:56:17,795 DEBUG [Thread-4043] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:56:17,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:56:17,797 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:56:17,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:56:17,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:56:17,799 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:56:17,799 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:56:17,799 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:56:17,799 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:56:17,799 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:17,800 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:17,801 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:56:17,801 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:17,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:56:17,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:56:17,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:17,802 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:17,802 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:56:17,802 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:17,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:56:17,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:17,802 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:56:17,803 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:56:17,803 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:56:17,803 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:56:17,803 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:56:17,803 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:56:17,804 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:56:17,804 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:17,804 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:17,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57236 deadline: 1681937837799, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:17,804 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. started... 2023-04-19 20:56:17,804 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:56:17,804 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. as already flushing 2023-04-19 20:56:17,804 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:17,804 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:56:17,804 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:56:17,804 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:17,804 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:56:17,805 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:56:17,806 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:17,806 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:17,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57246 deadline: 1681937837802, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:17,806 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:17,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:17,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:17,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:17,807 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:17,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:17,807 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:56:17,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:17,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:17,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:17,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:17,808 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:17,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57226 deadline: 1681937837804, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:17,808 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:56:17,808 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@dbfa1ba[Count = 0] remaining members to acquire global barrier 2023-04-19 20:56:17,808 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:56:17,808 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:17,811 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:17,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:17,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:17,811 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:56:17,811 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:56:17,811 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:56:17,811 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:17,811 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:56:17,812 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:17,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:17,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:17,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:17,812 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:56:17,813 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:56:17,813 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:56:17,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:17,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:17,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:17,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:17,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:17,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:17,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:17,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:56:17,815 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:56:17,815 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:56:17,815 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:56:17,815 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:56:17,815 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:56:17,815 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:56:17,816 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:17,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:17,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:17,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:17,817 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:17,817 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:56:17,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:17,817 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:17,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:17,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:17,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:56:17,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:56:17,818 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:17,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:17,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:17,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:17,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:17,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:17,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:17,824 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:17,824 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:17,824 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:17,824 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:17,824 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:56:17,824 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:56:17,824 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:17,824 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:17,824 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:17,824 DEBUG [Thread-4043] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:56:17,824 DEBUG [Thread-4043] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:56:17,824 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:56:17,824 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:56:17,824 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:56:17,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:56:17,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:56:17,825 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:56:17,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:56:17,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:56:17,906 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:17,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57236 deadline: 1681937837905, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:17,909 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:17,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57246 deadline: 1681937837907, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:17,911 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:17,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57226 deadline: 1681937837909, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:17,924 DEBUG [Thread-4043] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:56:17,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:56:17,927 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:56:17,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:56:17,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:56:17,928 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:56:17,928 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:56:17,928 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:56:17,928 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:56:17,928 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:17,929 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:17,933 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:56:17,933 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:17,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:56:17,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:56:17,933 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:17,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:17,933 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:56:17,934 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:17,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:56:17,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:17,934 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:56:17,934 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:56:17,934 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:56:17,934 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:56:17,934 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:56:17,934 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:56:17,935 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:56:17,935 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:17,935 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:56:17,935 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. started... 2023-04-19 20:56:17,935 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. as already flushing 2023-04-19 20:56:17,935 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:17,935 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:56:17,935 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:56:17,935 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:17,935 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:56:17,935 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:56:17,937 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:17,937 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:17,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:17,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:17,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:17,937 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:17,937 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:56:17,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:17,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:17,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:17,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:17,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:17,939 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:56:17,939 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2de9fcfe[Count = 0] remaining members to acquire global barrier 2023-04-19 20:56:17,939 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:56:17,939 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:17,941 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:17,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:17,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:17,941 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:56:17,941 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:56:17,941 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:17,941 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:56:17,941 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:56:17,943 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:56:17,943 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:56:17,943 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:17,943 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:56:17,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:17,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:17,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:17,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:17,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:17,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:17,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:17,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:17,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:17,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:17,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:56:17,947 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:56:17,947 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:56:17,947 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:56:17,947 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:56:17,947 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:56:17,947 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:56:17,949 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:17,949 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:17,949 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:56:17,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:17,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:17,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:17,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:17,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:17,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:56:17,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:56:17,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:17,950 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:17,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:17,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:17,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:17,951 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:17,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:17,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:17,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:17,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:17,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:17,958 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:56:17,958 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:56:17,958 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:56:17,959 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:56:17,959 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:56:17,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:56:17,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:56:17,959 DEBUG [Thread-4043] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:56:17,959 DEBUG [Thread-4043] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:56:17,959 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:17,959 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:17,959 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:17,959 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:17,959 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:17,959 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:17,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:56:17,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:56:17,959 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:17,960 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:56:17,961 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/A/837d84599682422d90be5bddf2164f89 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/837d84599682422d90be5bddf2164f89 2023-04-19 20:56:17,965 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 6873ba93789e63dd9f14391208147267/A of 6873ba93789e63dd9f14391208147267 into 837d84599682422d90be5bddf2164f89(size=26.1 K), total size for store is 51.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 20:56:17,965 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6873ba93789e63dd9f14391208147267: 2023-04-19 20:56:17,965 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267., storeName=6873ba93789e63dd9f14391208147267/A, priority=11, startTime=1681937777505; duration=0sec 2023-04-19 20:56:17,966 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 20:56:18,059 DEBUG [Thread-4043] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:56:18,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:56:18,061 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:56:18,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:56:18,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:56:18,063 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:56:18,063 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:56:18,063 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:56:18,063 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:56:18,064 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:18,064 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:18,065 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:56:18,065 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:18,065 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:56:18,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:56:18,065 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:18,065 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:56:18,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:18,066 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:18,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:56:18,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:18,066 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:56:18,067 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:56:18,067 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:56:18,067 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:56:18,067 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:56:18,067 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:56:18,067 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:56:18,067 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:56:18,067 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:18,067 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. started... 2023-04-19 20:56:18,067 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. as already flushing 2023-04-19 20:56:18,067 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:18,067 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:56:18,067 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:56:18,067 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:18,067 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:56:18,067 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:56:18,070 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:18,070 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:18,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:18,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:18,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:18,071 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:18,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:18,071 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:56:18,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:18,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:18,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:18,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:18,073 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:56:18,073 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f867cc7[Count = 0] remaining members to acquire global barrier 2023-04-19 20:56:18,073 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:56:18,073 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:18,074 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:18,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:18,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:18,074 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:56:18,075 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:56:18,075 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:56:18,075 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:18,075 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:56:18,076 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:18,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:18,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:18,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:18,076 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:56:18,076 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:56:18,076 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:56:18,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:18,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:18,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:18,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:18,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:18,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:18,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:18,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:56:18,079 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:56:18,079 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:56:18,079 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:56:18,079 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:56:18,079 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:56:18,079 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:56:18,081 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:18,081 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:18,081 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:18,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:18,081 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:56:18,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:18,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:18,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:18,081 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:56:18,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:56:18,081 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:18,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:18,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:18,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:18,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:18,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:18,083 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:18,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:18,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:18,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:18,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:18,087 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:56:18,087 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:56:18,087 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:56:18,087 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:56:18,087 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:56:18,087 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:56:18,087 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:18,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:56:18,087 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:18,087 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:18,087 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:18,087 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:18,087 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:18,087 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:18,087 DEBUG [Thread-4043] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:56:18,087 DEBUG [Thread-4043] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:56:18,087 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:56:18,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:56:18,108 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:18,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57236 deadline: 1681937838108, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:18,112 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:18,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57246 deadline: 1681937838110, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:18,116 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:18,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57226 deadline: 1681937838113, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:18,141 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041932c84ed833d1437688fa355654a3b166 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041932c84ed833d1437688fa355654a3b166 2023-04-19 20:56:18,146 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=325, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/A/e32da98b4c9f4f4194b62ee10e46e168 2023-04-19 20:56:18,154 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=325 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/B/33dc379f125042e7aa81de61a62d3f05 2023-04-19 20:56:18,162 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=325 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/C/7dabfe6513b64939b4d64f5432d2a4dc 2023-04-19 20:56:18,166 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/A/e32da98b4c9f4f4194b62ee10e46e168 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/e32da98b4c9f4f4194b62ee10e46e168 2023-04-19 20:56:18,169 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/e32da98b4c9f4f4194b62ee10e46e168, entries=300, sequenceid=325, filesize=46.1 K 2023-04-19 20:56:18,170 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/B/33dc379f125042e7aa81de61a62d3f05 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/33dc379f125042e7aa81de61a62d3f05 2023-04-19 20:56:18,173 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/33dc379f125042e7aa81de61a62d3f05, entries=150, sequenceid=325, filesize=11.9 K 2023-04-19 20:56:18,174 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/C/7dabfe6513b64939b4d64f5432d2a4dc as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/7dabfe6513b64939b4d64f5432d2a4dc 2023-04-19 20:56:18,177 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/7dabfe6513b64939b4d64f5432d2a4dc, entries=150, sequenceid=325, filesize=11.9 K 2023-04-19 20:56:18,177 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 6873ba93789e63dd9f14391208147267 in 454ms, sequenceid=325, compaction requested=true 2023-04-19 20:56:18,177 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 6873ba93789e63dd9f14391208147267: 2023-04-19 20:56:18,178 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 20:56:18,178 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-19 20:56:18,178 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-19 20:56:18,178 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 20:56:18,178 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 20:56:18,179 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37240 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-19 20:56:18,179 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1912): 6873ba93789e63dd9f14391208147267/B is initiating minor compaction (all files) 2023-04-19 20:56:18,179 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6873ba93789e63dd9f14391208147267/B in TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:18,179 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/6b7e2a62e336464d9940c09feb96e14f, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/9750333ee29047d2af6c880445330aeb, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/33dc379f125042e7aa81de61a62d3f05] into tmpdir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp, totalSize=36.4 K 2023-04-19 20:56:18,179 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 99944 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-19 20:56:18,179 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1912): 6873ba93789e63dd9f14391208147267/A is initiating minor compaction (all files) 2023-04-19 20:56:18,179 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 6873ba93789e63dd9f14391208147267/A in TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:18,179 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 6b7e2a62e336464d9940c09feb96e14f, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=294, earliestPutTs=1681937776732 2023-04-19 20:56:18,179 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/837d84599682422d90be5bddf2164f89, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/607f4960866b4b768437363b25dd3849, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/e32da98b4c9f4f4194b62ee10e46e168] into tmpdir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp, totalSize=97.6 K 2023-04-19 20:56:18,180 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 9750333ee29047d2af6c880445330aeb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=315, earliestPutTs=1681937777070 2023-04-19 20:56:18,180 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 837d84599682422d90be5bddf2164f89, keycount=150, bloomtype=ROW, size=26.1 K, encoding=NONE, compression=NONE, seqNum=294, earliestPutTs=1681937776732 2023-04-19 20:56:18,180 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 33dc379f125042e7aa81de61a62d3f05, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=325, earliestPutTs=1681937777708 2023-04-19 20:56:18,180 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 607f4960866b4b768437363b25dd3849, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=315, earliestPutTs=1681937777070 2023-04-19 20:56:18,181 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting e32da98b4c9f4f4194b62ee10e46e168, keycount=300, bloomtype=ROW, size=46.1 K, encoding=NONE, compression=NONE, seqNum=325, earliestPutTs=1681937777707 2023-04-19 20:56:18,187 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] throttle.PressureAwareThroughputController(145): 6873ba93789e63dd9f14391208147267#B#compaction#611 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-19 20:56:18,187 DEBUG [Thread-4043] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:56:18,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:56:18,190 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:56:18,190 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] throttle.PressureAwareThroughputController(145): 6873ba93789e63dd9f14391208147267#A#compaction#612 average throughput is 4.93 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 20:56:18,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:56:18,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:56:18,191 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:56:18,191 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:56:18,191 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:56:18,191 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:56:18,191 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:18,191 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:18,193 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:56:18,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:56:18,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:56:18,193 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:18,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:18,194 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:18,194 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:56:18,194 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:18,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:56:18,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:18,195 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:56:18,200 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:56:18,200 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:56:18,200 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:56:18,200 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:56:18,200 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:56:18,200 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:56:18,200 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:56:18,200 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:18,200 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. started... 2023-04-19 20:56:18,201 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6873ba93789e63dd9f14391208147267 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-19 20:56:18,201 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6873ba93789e63dd9f14391208147267, store=A 2023-04-19 20:56:18,201 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 20:56:18,201 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6873ba93789e63dd9f14391208147267, store=B 2023-04-19 20:56:18,201 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 20:56:18,201 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6873ba93789e63dd9f14391208147267, store=C 2023-04-19 20:56:18,201 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 20:56:18,211 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/B/7693a11c4fa743a5a26c8726ef23af9c as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/7693a11c4fa743a5a26c8726ef23af9c 2023-04-19 20:56:18,216 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 6873ba93789e63dd9f14391208147267/B of 6873ba93789e63dd9f14391208147267 into 7693a11c4fa743a5a26c8726ef23af9c(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 20:56:18,216 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6873ba93789e63dd9f14391208147267: 2023-04-19 20:56:18,216 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267., storeName=6873ba93789e63dd9f14391208147267/B, priority=13, startTime=1681937778178; duration=0sec 2023-04-19 20:56:18,216 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 20:56:18,216 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-19 20:56:18,217 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37206 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-19 20:56:18,217 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1912): 6873ba93789e63dd9f14391208147267/C is initiating minor compaction (all files) 2023-04-19 20:56:18,217 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6873ba93789e63dd9f14391208147267/C in TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:18,217 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/caf2f2e21ba949b2bbd08822eff3eb91, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/df3c8da2747646e589d8c282d19e756b, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/7dabfe6513b64939b4d64f5432d2a4dc] into tmpdir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp, totalSize=36.3 K 2023-04-19 20:56:18,218 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting caf2f2e21ba949b2bbd08822eff3eb91, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=294, earliestPutTs=1681937776732 2023-04-19 20:56:18,218 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting df3c8da2747646e589d8c282d19e756b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=315, earliestPutTs=1681937777070 2023-04-19 20:56:18,218 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 7dabfe6513b64939b4d64f5432d2a4dc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=325, earliestPutTs=1681937777708 2023-04-19 20:56:18,240 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] throttle.PressureAwareThroughputController(145): 6873ba93789e63dd9f14391208147267#C#compaction#614 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 20:56:18,245 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304192a49aa7926c44f15a2fcd3232a42781f to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304192a49aa7926c44f15a2fcd3232a42781f 2023-04-19 20:56:18,252 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/A/deb0d77a660046108ba1ce6d3d9e3396 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/deb0d77a660046108ba1ce6d3d9e3396 2023-04-19 20:56:18,256 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 6873ba93789e63dd9f14391208147267/A of 6873ba93789e63dd9f14391208147267 into deb0d77a660046108ba1ce6d3d9e3396(size=26.2 K), total size for store is 26.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 20:56:18,257 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6873ba93789e63dd9f14391208147267: 2023-04-19 20:56:18,257 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267., storeName=6873ba93789e63dd9f14391208147267/A, priority=13, startTime=1681937778178; duration=0sec 2023-04-19 20:56:18,257 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 20:56:18,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(9158): Flush requested on 6873ba93789e63dd9f14391208147267 2023-04-19 20:56:18,413 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. as already flushing 2023-04-19 20:56:18,423 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:18,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57246 deadline: 1681937838421, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:18,428 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:18,428 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:18,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57226 deadline: 1681937838424, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:18,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57236 deadline: 1681937838424, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:18,525 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:18,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57246 deadline: 1681937838525, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:18,529 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:18,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57226 deadline: 1681937838529, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:18,530 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:18,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57236 deadline: 1681937838529, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:18,650 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/C/826a0a4407cb488db170becc2217b51c as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/826a0a4407cb488db170becc2217b51c 2023-04-19 20:56:18,656 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=363, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/A/b5728a88b2374867a4f43c817e80f21b 2023-04-19 20:56:18,660 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 6873ba93789e63dd9f14391208147267/C of 6873ba93789e63dd9f14391208147267 into 826a0a4407cb488db170becc2217b51c(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 20:56:18,660 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6873ba93789e63dd9f14391208147267: 2023-04-19 20:56:18,660 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267., storeName=6873ba93789e63dd9f14391208147267/C, priority=13, startTime=1681937778178; duration=0sec 2023-04-19 20:56:18,660 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 20:56:18,669 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=363 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/B/ec7bd190818e4214b6d960159073885b 2023-04-19 20:56:18,731 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:18,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57246 deadline: 1681937838727, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:18,733 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:18,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57236 deadline: 1681937838730, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:18,735 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:18,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57226 deadline: 1681937838731, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:19,033 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:19,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57246 deadline: 1681937839032, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:19,037 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:19,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57236 deadline: 1681937839035, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:19,039 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:19,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57226 deadline: 1681937839037, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:19,078 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=363 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/C/e3c72c0402434b0d999f0f6483a108d9 2023-04-19 20:56:19,082 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/A/b5728a88b2374867a4f43c817e80f21b as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/b5728a88b2374867a4f43c817e80f21b 2023-04-19 20:56:19,085 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/b5728a88b2374867a4f43c817e80f21b, entries=150, sequenceid=363, filesize=25.4 K 2023-04-19 20:56:19,085 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/B/ec7bd190818e4214b6d960159073885b as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/ec7bd190818e4214b6d960159073885b 2023-04-19 20:56:19,089 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/ec7bd190818e4214b6d960159073885b, entries=150, sequenceid=363, filesize=11.9 K 2023-04-19 20:56:19,089 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/C/e3c72c0402434b0d999f0f6483a108d9 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/e3c72c0402434b0d999f0f6483a108d9 2023-04-19 20:56:19,092 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/e3c72c0402434b0d999f0f6483a108d9, entries=150, sequenceid=363, filesize=11.9 K 2023-04-19 20:56:19,093 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=53.67 KB/54960 for 6873ba93789e63dd9f14391208147267 in 892ms, sequenceid=363, compaction requested=false 2023-04-19 20:56:19,093 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6873ba93789e63dd9f14391208147267: 2023-04-19 20:56:19,093 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:19,093 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:56:19,093 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:56:19,093 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:19,093 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:56:19,093 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:56:19,095 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:19,095 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:19,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:19,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:19,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:19,095 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:19,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:19,095 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:56:19,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:19,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:19,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:19,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:19,097 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:56:19,097 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@118a38c2[Count = 0] remaining members to acquire global barrier 2023-04-19 20:56:19,097 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:56:19,097 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:19,098 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:19,098 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:19,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:19,098 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:56:19,098 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:56:19,098 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:56:19,098 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:19,098 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:56:19,102 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:19,102 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:56:19,102 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:56:19,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:19,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:19,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:19,102 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:56:19,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:19,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:19,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:19,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:19,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:19,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:19,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:19,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:56:19,105 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:56:19,105 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:56:19,105 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:56:19,105 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:56:19,105 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:56:19,105 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:56:19,107 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:19,107 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:19,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:19,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:19,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:19,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:19,107 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:56:19,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:19,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:19,107 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:19,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:56:19,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:56:19,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:19,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:19,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:19,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:19,109 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:19,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:19,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:19,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:19,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:19,111 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:56:19,111 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:19,111 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:19,111 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:19,111 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:19,111 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:19,111 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:19,111 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:19,111 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:56:19,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:56:19,111 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:56:19,111 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:56:19,111 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:56:19,111 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:56:19,111 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:56:19,112 DEBUG [Thread-4043] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:56:19,112 DEBUG [Thread-4043] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:56:19,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:56:19,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:56:19,212 DEBUG [Thread-4043] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:56:19,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:56:19,214 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:56:19,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:56:19,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:56:19,215 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:56:19,215 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:56:19,215 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:56:19,215 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:56:19,216 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:19,216 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:19,218 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:56:19,218 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:19,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:56:19,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:56:19,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:19,218 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:19,219 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:56:19,219 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:19,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:56:19,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:19,219 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:56:19,219 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:56:19,219 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:56:19,219 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:56:19,220 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:56:19,220 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:56:19,220 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:56:19,220 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:56:19,220 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:19,220 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. started... 2023-04-19 20:56:19,220 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6873ba93789e63dd9f14391208147267 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-19 20:56:19,220 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6873ba93789e63dd9f14391208147267, store=A 2023-04-19 20:56:19,220 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:56:19,220 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6873ba93789e63dd9f14391208147267, store=B 2023-04-19 20:56:19,220 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:56:19,221 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6873ba93789e63dd9f14391208147267, store=C 2023-04-19 20:56:19,221 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:56:19,233 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230419635ad6cc96244b768208a0cce991a482 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419635ad6cc96244b768208a0cce991a482 2023-04-19 20:56:19,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(9158): Flush requested on 6873ba93789e63dd9f14391208147267 2023-04-19 20:56:19,307 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. as already flushing 2023-04-19 20:56:19,337 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681937779321, max timestamp=1681937779335Num uniques 859; 2023-04-19 20:56:19,337 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681937779321, max timestamp=1681937779335Num uniques 859; 2023-04-19 20:56:19,337 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681937779321, max timestamp=1681937779335Num uniques 859; 2023-04-19 20:56:19,337 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 20:56:19,337 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:19,337 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 20:56:19,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57204 deadline: 1681937839334, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:19,337 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 20:56:19,380 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:19,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57210 deadline: 1681937839378, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:19,440 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:19,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57204 deadline: 1681937839438, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:19,483 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:19,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57210 deadline: 1681937839482, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:19,540 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:19,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57246 deadline: 1681937839538, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:19,541 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:19,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57236 deadline: 1681937839540, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:19,544 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:19,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57226 deadline: 1681937839541, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:19,642 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=377, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/A/c2c5846bd82740d0ac3d4ad8028faf3f 2023-04-19 20:56:19,643 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:19,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57204 deadline: 1681937839642, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:19,688 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:19,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57210 deadline: 1681937839685, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:19,947 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:19,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57204 deadline: 1681937839945, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:19,992 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:19,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57210 deadline: 1681937839990, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:20,054 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=377 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/B/a7e715205f8249a7a64057eefe5d3f47 2023-04-19 20:56:20,068 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=377 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/C/4f6b7a4ac90844378013533328745a19 2023-04-19 20:56:20,072 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/A/c2c5846bd82740d0ac3d4ad8028faf3f as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/c2c5846bd82740d0ac3d4ad8028faf3f 2023-04-19 20:56:20,075 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/c2c5846bd82740d0ac3d4ad8028faf3f, entries=150, sequenceid=377, filesize=25.4 K 2023-04-19 20:56:20,076 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/B/a7e715205f8249a7a64057eefe5d3f47 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/a7e715205f8249a7a64057eefe5d3f47 2023-04-19 20:56:20,079 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/a7e715205f8249a7a64057eefe5d3f47, entries=150, sequenceid=377, filesize=11.9 K 2023-04-19 20:56:20,080 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/C/4f6b7a4ac90844378013533328745a19 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/4f6b7a4ac90844378013533328745a19 2023-04-19 20:56:20,082 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/4f6b7a4ac90844378013533328745a19, entries=150, sequenceid=377, filesize=11.9 K 2023-04-19 20:56:20,083 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 6873ba93789e63dd9f14391208147267 in 863ms, sequenceid=377, compaction requested=true 2023-04-19 20:56:20,083 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6873ba93789e63dd9f14391208147267: 2023-04-19 20:56:20,083 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:20,083 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:56:20,083 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:56:20,083 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:20,083 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:56:20,083 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:56:20,086 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:20,086 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:20,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:20,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:20,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:20,086 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:20,086 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:56:20,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:20,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:20,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:20,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:20,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:20,088 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:56:20,088 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@61c8e938[Count = 0] remaining members to acquire global barrier 2023-04-19 20:56:20,088 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:56:20,088 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:20,089 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:20,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:20,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:20,089 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:56:20,089 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:56:20,089 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:56:20,089 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:20,089 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:56:20,091 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:56:20,091 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:20,091 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:56:20,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:20,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:20,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:20,091 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:56:20,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:20,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:20,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:20,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:20,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:20,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:20,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:20,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:56:20,094 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:56:20,094 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:56:20,094 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:56:20,094 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:56:20,094 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:56:20,094 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:56:20,096 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:20,096 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:56:20,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:20,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:20,096 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:20,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:20,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:20,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:20,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:20,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:56:20,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:56:20,096 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:20,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:20,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:20,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:20,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:20,098 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:20,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:20,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:20,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:20,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:20,101 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:20,101 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:20,101 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:56:20,102 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:56:20,102 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:56:20,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:56:20,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:56:20,102 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:56:20,102 DEBUG [Thread-4043] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:56:20,101 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:56:20,101 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:56:20,101 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:20,102 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:20,102 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:20,102 DEBUG [Thread-4043] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:56:20,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:56:20,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:56:20,102 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:20,102 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:20,202 DEBUG [Thread-4043] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:56:20,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:56:20,204 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:56:20,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:56:20,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:56:20,206 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:56:20,206 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:56:20,206 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:56:20,206 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:56:20,207 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:20,207 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:20,208 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:56:20,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:56:20,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:56:20,208 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:20,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:20,208 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:20,208 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:56:20,209 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:20,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:56:20,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:20,209 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:56:20,209 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:56:20,209 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:56:20,210 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:56:20,210 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:56:20,210 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:56:20,210 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:56:20,210 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:56:20,210 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:20,210 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. started... 2023-04-19 20:56:20,210 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6873ba93789e63dd9f14391208147267 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-04-19 20:56:20,210 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6873ba93789e63dd9f14391208147267, store=A 2023-04-19 20:56:20,211 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 20:56:20,211 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6873ba93789e63dd9f14391208147267, store=B 2023-04-19 20:56:20,211 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 20:56:20,211 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6873ba93789e63dd9f14391208147267, store=C 2023-04-19 20:56:20,211 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 20:56:20,228 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230419ac03677948bd4dda91289b467ce91eb9 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419ac03677948bd4dda91289b467ce91eb9 2023-04-19 20:56:20,234 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=413, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/A/579c2d90b9a14427aa382495e3b7e88c 2023-04-19 20:56:20,243 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=413 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/B/eaf7d535a1fa4869ba67786952e0b4d1 2023-04-19 20:56:20,252 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=413 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/C/11f608a8f60f486a8094a7304764ae55 2023-04-19 20:56:20,256 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/A/579c2d90b9a14427aa382495e3b7e88c as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/579c2d90b9a14427aa382495e3b7e88c 2023-04-19 20:56:20,259 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/579c2d90b9a14427aa382495e3b7e88c, entries=150, sequenceid=413, filesize=25.4 K 2023-04-19 20:56:20,260 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/B/eaf7d535a1fa4869ba67786952e0b4d1 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/eaf7d535a1fa4869ba67786952e0b4d1 2023-04-19 20:56:20,263 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/eaf7d535a1fa4869ba67786952e0b4d1, entries=150, sequenceid=413, filesize=11.9 K 2023-04-19 20:56:20,264 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/C/11f608a8f60f486a8094a7304764ae55 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/11f608a8f60f486a8094a7304764ae55 2023-04-19 20:56:20,267 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/11f608a8f60f486a8094a7304764ae55, entries=150, sequenceid=413, filesize=11.9 K 2023-04-19 20:56:20,267 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=0 B/0 for 6873ba93789e63dd9f14391208147267 in 57ms, sequenceid=413, compaction requested=true 2023-04-19 20:56:20,267 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6873ba93789e63dd9f14391208147267: 2023-04-19 20:56:20,267 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:20,268 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:56:20,268 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:56:20,268 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:20,268 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:56:20,268 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:56:20,269 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:20,269 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:20,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:20,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:20,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:20,270 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:20,270 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:56:20,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:20,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:20,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:20,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:20,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:20,272 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:56:20,272 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@324e7f3[Count = 0] remaining members to acquire global barrier 2023-04-19 20:56:20,272 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:56:20,272 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:20,273 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:20,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:20,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:20,273 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:56:20,273 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:56:20,273 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:56:20,273 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:20,273 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:56:20,275 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:56:20,276 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:56:20,276 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:20,276 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:56:20,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:20,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:20,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:20,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:20,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:20,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:20,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:20,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:20,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:20,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:20,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:56:20,279 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:56:20,279 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:56:20,279 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:56:20,279 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:56:20,279 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:56:20,279 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:56:20,281 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:20,281 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:20,281 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:20,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:20,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:20,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:20,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:20,281 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:56:20,281 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:56:20,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:56:20,281 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:20,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:20,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:20,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:20,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:20,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:20,283 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:20,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:20,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:20,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:20,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:20,285 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:20,285 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:56:20,285 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:20,285 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:20,285 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:20,285 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:56:20,285 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:20,285 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:20,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:56:20,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:56:20,285 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:56:20,285 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:20,285 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:56:20,286 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:56:20,286 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:56:20,286 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:56:20,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:56:20,286 DEBUG [Thread-4043] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:56:20,286 DEBUG [Thread-4043] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:56:20,386 DEBUG [Thread-4043] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:56:20,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:56:20,388 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:56:20,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:56:20,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:56:20,389 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:56:20,389 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:56:20,389 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:56:20,390 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:56:20,390 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:20,390 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:20,391 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:56:20,391 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:20,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:56:20,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:56:20,391 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:20,392 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:56:20,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:20,392 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:20,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:56:20,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:20,392 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:56:20,393 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:56:20,393 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:56:20,393 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:56:20,393 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:56:20,393 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:56:20,393 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:56:20,393 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:20,393 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:56:20,393 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. started... 2023-04-19 20:56:20,393 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6873ba93789e63dd9f14391208147267: 2023-04-19 20:56:20,393 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:20,393 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:56:20,394 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:56:20,394 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:20,394 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:56:20,394 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:56:20,396 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:20,396 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:20,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:20,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:20,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:20,397 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:20,397 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:56:20,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:20,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:20,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:20,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:20,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:20,398 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:56:20,398 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4f08d314[Count = 0] remaining members to acquire global barrier 2023-04-19 20:56:20,398 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:56:20,398 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:20,399 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:20,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:20,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:20,400 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:56:20,400 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:56:20,400 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:20,400 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:56:20,400 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:56:20,401 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:20,401 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:56:20,401 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:56:20,401 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:56:20,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:20,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:20,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:20,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:20,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:20,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:20,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:20,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:20,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:20,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:20,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:56:20,404 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:56:20,404 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:56:20,404 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:56:20,404 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:56:20,404 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:56:20,404 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:56:20,406 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:20,406 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:20,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:20,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:20,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:20,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:20,406 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:56:20,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:20,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:56:20,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:56:20,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:20,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:20,407 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:20,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:20,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:20,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:20,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:20,408 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:20,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:20,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:20,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:20,412 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:56:20,412 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:20,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:56:20,412 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:56:20,412 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:56:20,412 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:56:20,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:56:20,412 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:56:20,412 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:20,412 DEBUG [Thread-4043] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:56:20,412 DEBUG [Thread-4043] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:56:20,412 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:20,412 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:20,413 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:20,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:56:20,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:56:20,413 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:20,413 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:20,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(9158): Flush requested on 6873ba93789e63dd9f14391208147267 2023-04-19 20:56:20,505 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 6873ba93789e63dd9f14391208147267 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-19 20:56:20,506 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6873ba93789e63dd9f14391208147267, store=A 2023-04-19 20:56:20,506 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:56:20,506 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6873ba93789e63dd9f14391208147267, store=B 2023-04-19 20:56:20,506 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:56:20,506 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6873ba93789e63dd9f14391208147267, store=C 2023-04-19 20:56:20,506 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:56:20,513 DEBUG [Thread-4043] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:56:20,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:56:20,515 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:56:20,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:56:20,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:56:20,518 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:56:20,519 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:56:20,519 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:56:20,519 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:56:20,519 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:20,519 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:20,523 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:56:20,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:56:20,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:56:20,524 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:20,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:20,524 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:20,524 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:56:20,524 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:20,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:56:20,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:20,525 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:56:20,525 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:56:20,525 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:56:20,525 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:56:20,525 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:56:20,525 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:56:20,525 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:56:20,525 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:56:20,525 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:20,525 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. started... 2023-04-19 20:56:20,525 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. as already flushing 2023-04-19 20:56:20,525 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:20,525 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:56:20,525 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:56:20,526 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:20,526 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:56:20,526 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:56:20,527 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:20,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:20,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:20,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:20,527 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:20,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:20,528 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:20,528 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:56:20,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:20,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:20,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:20,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:20,529 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:56:20,529 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@f84ae1f[Count = 0] remaining members to acquire global barrier 2023-04-19 20:56:20,529 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:56:20,529 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:20,532 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:20,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:20,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:20,532 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:56:20,532 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:56:20,532 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:20,532 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:56:20,532 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:56:20,533 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:20,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:20,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:20,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:20,534 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:56:20,534 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:56:20,534 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:56:20,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:20,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:20,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:20,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:20,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:20,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:20,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:20,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:56:20,536 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:56:20,536 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:56:20,536 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:56:20,536 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:56:20,536 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:56:20,536 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:56:20,538 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:20,538 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:20,538 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:56:20,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:20,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:20,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:20,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:20,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:20,538 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:20,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:20,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:56:20,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:56:20,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:20,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:20,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:20,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:20,539 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:20,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:20,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:20,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:20,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:20,542 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:56:20,542 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:56:20,542 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:56:20,542 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:20,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:56:20,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:56:20,542 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:20,543 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:20,543 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:20,543 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:20,543 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:20,543 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:20,542 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:56:20,543 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:56:20,543 DEBUG [Thread-4043] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:56:20,543 DEBUG [Thread-4043] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:56:20,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:56:20,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:56:20,543 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:56:20,549 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681937780537, max timestamp=1681937780547Num uniques 859; 2023-04-19 20:56:20,549 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 20:56:20,549 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681937780537, max timestamp=1681937780547Num uniques 859; 2023-04-19 20:56:20,549 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 20:56:20,549 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681937780537, max timestamp=1681937780547Num uniques 859; 2023-04-19 20:56:20,549 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 20:56:20,570 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681937780561, max timestamp=1681937780568Num uniques 1018; 2023-04-19 20:56:20,570 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-19 20:56:20,570 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681937780561, max timestamp=1681937780568Num uniques 1018; 2023-04-19 20:56:20,571 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-19 20:56:20,571 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681937780561, max timestamp=1681937780568Num uniques 1018; 2023-04-19 20:56:20,571 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-19 20:56:20,587 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:20,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57246 deadline: 1681937840582, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:20,589 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:20,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57210 deadline: 1681937840585, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:20,589 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:20,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57204 deadline: 1681937840585, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:20,591 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:20,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57236 deadline: 1681937840586, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:20,591 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:20,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57226 deadline: 1681937840588, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:20,643 DEBUG [Thread-4043] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:56:20,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:56:20,645 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:56:20,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:56:20,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:56:20,647 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:56:20,647 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:56:20,647 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:56:20,647 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:56:20,647 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:20,647 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:20,649 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:56:20,649 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:20,649 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:56:20,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:56:20,649 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:20,649 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:56:20,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:20,650 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:20,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:56:20,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:20,650 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:56:20,650 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:56:20,651 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:56:20,651 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:56:20,651 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:56:20,651 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:56:20,651 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:56:20,651 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:56:20,651 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:20,651 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. started... 2023-04-19 20:56:20,651 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. as already flushing 2023-04-19 20:56:20,651 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:20,651 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:56:20,651 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:56:20,651 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:20,651 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:56:20,651 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:56:20,653 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:20,653 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:20,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:20,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:20,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:20,653 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:20,653 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:56:20,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:20,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:20,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:20,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:20,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:20,655 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:56:20,655 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@292307fe[Count = 0] remaining members to acquire global barrier 2023-04-19 20:56:20,655 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:56:20,655 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:20,656 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:20,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:20,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:20,656 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:56:20,656 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:56:20,656 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:56:20,656 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:20,656 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:56:20,657 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:20,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:20,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:20,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:20,658 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:56:20,658 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:56:20,658 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:56:20,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:20,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:20,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:20,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:20,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:20,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:20,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:20,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:56:20,660 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:56:20,660 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:56:20,660 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:56:20,660 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:56:20,660 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:56:20,660 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:56:20,662 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:20,662 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:20,662 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:56:20,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:20,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:20,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:20,662 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:20,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:20,663 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:56:20,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:56:20,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:20,663 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:20,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:20,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:20,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:20,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:20,664 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:20,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:20,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:20,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:20,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:20,667 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:20,667 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:20,667 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:56:20,667 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:56:20,667 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:56:20,667 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:56:20,667 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:20,667 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:20,667 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:56:20,667 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:56:20,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:56:20,667 DEBUG [Thread-4043] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:56:20,667 DEBUG [Thread-4043] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:56:20,667 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:56:20,667 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:20,667 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:20,667 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:20,667 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:56:20,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:56:20,692 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:20,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57246 deadline: 1681937840688, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:20,693 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:20,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57210 deadline: 1681937840690, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:20,693 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:20,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57204 deadline: 1681937840690, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:20,693 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:20,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57226 deadline: 1681937840692, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:20,698 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:20,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57236 deadline: 1681937840693, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:20,767 DEBUG [Thread-4043] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:56:20,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:56:20,769 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:56:20,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:56:20,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:56:20,771 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:56:20,771 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:56:20,771 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:56:20,771 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:56:20,771 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:20,771 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:20,772 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:20,773 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:56:20,773 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:56:20,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:56:20,773 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:20,773 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:56:20,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:20,773 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:20,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:56:20,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:20,774 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:56:20,774 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:56:20,774 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:56:20,774 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:56:20,774 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:56:20,774 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:56:20,774 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:56:20,774 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:56:20,774 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:20,774 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. started... 2023-04-19 20:56:20,775 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. as already flushing 2023-04-19 20:56:20,775 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:20,775 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:56:20,775 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:56:20,775 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:20,775 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:56:20,775 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:56:20,776 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:20,776 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:20,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:20,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:20,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:20,777 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:20,777 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:56:20,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:20,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:20,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:20,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:20,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:20,781 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:56:20,781 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@d9c6656[Count = 0] remaining members to acquire global barrier 2023-04-19 20:56:20,781 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:56:20,781 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:20,783 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:20,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:20,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:20,783 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:20,783 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:56:20,783 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:56:20,783 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:56:20,784 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:56:20,785 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:20,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:20,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:20,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:20,785 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:56:20,785 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:56:20,785 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:56:20,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:20,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:20,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:20,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:20,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:20,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:20,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:20,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:56:20,787 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:56:20,787 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:56:20,787 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:56:20,787 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:56:20,787 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:56:20,787 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:56:20,789 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:20,789 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:20,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:20,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:20,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:20,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:20,789 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:56:20,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:20,789 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:20,790 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:56:20,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:56:20,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:20,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:20,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:20,791 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:20,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:20,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:20,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:20,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:20,793 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:20,793 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:20,793 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:20,793 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:20,793 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:20,793 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:20,793 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:20,793 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:56:20,793 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:56:20,794 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:56:20,794 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:56:20,800 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:56:20,800 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:56:20,800 DEBUG [Thread-4043] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:56:20,800 DEBUG [Thread-4043] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:56:20,800 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:56:20,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:56:20,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:56:20,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:56:20,897 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:20,897 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:20,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57246 deadline: 1681937840894, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:20,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57204 deadline: 1681937840894, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:20,897 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:20,898 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:20,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57210 deadline: 1681937840895, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:20,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57226 deadline: 1681937840894, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:20,900 DEBUG [Thread-4043] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:56:20,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:56:20,902 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:56:20,903 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:20,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57236 deadline: 1681937840899, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:20,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:56:20,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:56:20,904 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:56:20,904 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:56:20,904 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:56:20,904 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:56:20,904 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:20,904 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:20,907 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:56:20,907 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:20,907 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:56:20,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:56:20,908 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:20,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:20,908 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:56:20,908 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:20,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:56:20,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:20,909 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:56:20,909 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:56:20,909 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:56:20,909 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:56:20,909 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:56:20,909 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:56:20,909 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:56:20,909 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:56:20,909 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:20,909 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. started... 2023-04-19 20:56:20,909 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. as already flushing 2023-04-19 20:56:20,909 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:20,909 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:56:20,910 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:56:20,910 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:20,910 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:56:20,910 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:56:20,911 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:20,911 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:20,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:20,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:20,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:20,911 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:20,911 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:56:20,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:20,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:20,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:20,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:20,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:20,912 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:56:20,913 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4859482[Count = 0] remaining members to acquire global barrier 2023-04-19 20:56:20,913 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:56:20,913 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:20,916 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:20,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:20,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:20,916 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:56:20,916 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:56:20,916 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:20,916 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:56:20,916 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:56:20,921 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:56:20,921 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:56:20,921 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:56:20,921 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:20,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:20,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:20,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:20,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:20,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:20,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:20,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:20,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:20,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:20,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:20,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:56:20,924 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:56:20,924 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230419f042715ab0054a799a40c97a0ce69614 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419f042715ab0054a799a40c97a0ce69614 2023-04-19 20:56:20,924 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:56:20,925 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:56:20,925 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:56:20,925 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:56:20,925 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:56:20,926 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:20,926 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:56:20,926 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:20,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:20,927 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:20,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:20,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:20,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:20,927 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:56:20,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:56:20,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:20,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:20,927 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:20,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:20,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:20,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:20,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:20,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:20,929 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:20,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:20,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:20,930 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=424, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/A/11b3f4bb7946432ca51e9ceaa3667218 2023-04-19 20:56:20,932 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:20,932 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:20,932 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:20,932 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:20,932 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:20,932 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:20,932 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:20,932 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:56:20,932 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:56:20,932 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:56:20,932 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:56:20,932 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:56:20,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:56:20,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:56:20,932 DEBUG [Thread-4043] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:56:20,932 DEBUG [Thread-4043] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:56:20,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:56:20,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:56:20,940 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=424 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/B/9eaaa7d1ca274c85bbc7d332c7f43a0e 2023-04-19 20:56:20,948 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=424 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/C/0f6ae49ecfe74a13bb4bc05706a01e7d 2023-04-19 20:56:20,951 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/A/11b3f4bb7946432ca51e9ceaa3667218 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/11b3f4bb7946432ca51e9ceaa3667218 2023-04-19 20:56:20,955 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/11b3f4bb7946432ca51e9ceaa3667218, entries=250, sequenceid=424, filesize=39.2 K 2023-04-19 20:56:20,956 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/B/9eaaa7d1ca274c85bbc7d332c7f43a0e as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/9eaaa7d1ca274c85bbc7d332c7f43a0e 2023-04-19 20:56:20,960 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/9eaaa7d1ca274c85bbc7d332c7f43a0e, entries=150, sequenceid=424, filesize=11.9 K 2023-04-19 20:56:20,960 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/C/0f6ae49ecfe74a13bb4bc05706a01e7d as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/0f6ae49ecfe74a13bb4bc05706a01e7d 2023-04-19 20:56:20,963 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/0f6ae49ecfe74a13bb4bc05706a01e7d, entries=150, sequenceid=424, filesize=11.9 K 2023-04-19 20:56:20,964 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 6873ba93789e63dd9f14391208147267 in 459ms, sequenceid=424, compaction requested=true 2023-04-19 20:56:20,964 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 6873ba93789e63dd9f14391208147267: 2023-04-19 20:56:20,964 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 20:56:20,964 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 20:56:20,964 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-19 20:56:20,964 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-19 20:56:20,964 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 20:56:20,966 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61714 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-19 20:56:20,966 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1912): 6873ba93789e63dd9f14391208147267/B is initiating minor compaction (all files) 2023-04-19 20:56:20,966 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6873ba93789e63dd9f14391208147267/B in TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:20,966 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/7693a11c4fa743a5a26c8726ef23af9c, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/ec7bd190818e4214b6d960159073885b, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/a7e715205f8249a7a64057eefe5d3f47, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/eaf7d535a1fa4869ba67786952e0b4d1, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/9eaaa7d1ca274c85bbc7d332c7f43a0e] into tmpdir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp, totalSize=60.3 K 2023-04-19 20:56:20,966 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 145098 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-19 20:56:20,966 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1912): 6873ba93789e63dd9f14391208147267/A is initiating minor compaction (all files) 2023-04-19 20:56:20,966 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 6873ba93789e63dd9f14391208147267/A in TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:20,966 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/deb0d77a660046108ba1ce6d3d9e3396, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/b5728a88b2374867a4f43c817e80f21b, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/c2c5846bd82740d0ac3d4ad8028faf3f, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/579c2d90b9a14427aa382495e3b7e88c, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/11b3f4bb7946432ca51e9ceaa3667218] into tmpdir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp, totalSize=141.7 K 2023-04-19 20:56:20,967 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 7693a11c4fa743a5a26c8726ef23af9c, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=325, earliestPutTs=1681937777708 2023-04-19 20:56:20,967 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting deb0d77a660046108ba1ce6d3d9e3396, keycount=150, bloomtype=ROW, size=26.2 K, encoding=NONE, compression=NONE, seqNum=325, earliestPutTs=1681937777708 2023-04-19 20:56:20,967 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting ec7bd190818e4214b6d960159073885b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=363, earliestPutTs=1681937777795 2023-04-19 20:56:20,967 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting b5728a88b2374867a4f43c817e80f21b, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=363, earliestPutTs=1681937777795 2023-04-19 20:56:20,967 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting a7e715205f8249a7a64057eefe5d3f47, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=377, earliestPutTs=1681937778419 2023-04-19 20:56:20,967 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting c2c5846bd82740d0ac3d4ad8028faf3f, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=377, earliestPutTs=1681937778419 2023-04-19 20:56:20,967 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting eaf7d535a1fa4869ba67786952e0b4d1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=413, earliestPutTs=1681937779358 2023-04-19 20:56:20,968 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 579c2d90b9a14427aa382495e3b7e88c, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=413, earliestPutTs=1681937779358 2023-04-19 20:56:20,968 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 9eaaa7d1ca274c85bbc7d332c7f43a0e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=424, earliestPutTs=1681937780486 2023-04-19 20:56:20,968 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 11b3f4bb7946432ca51e9ceaa3667218, keycount=250, bloomtype=ROW, size=39.2 K, encoding=NONE, compression=NONE, seqNum=424, earliestPutTs=1681937780476 2023-04-19 20:56:20,976 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] throttle.PressureAwareThroughputController(145): 6873ba93789e63dd9f14391208147267#B#compaction#626 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 20:56:20,989 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] throttle.PressureAwareThroughputController(145): 6873ba93789e63dd9f14391208147267#A#compaction#627 average throughput is 1.79 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 20:56:21,032 DEBUG [Thread-4043] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:56:21,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:56:21,034 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:56:21,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:56:21,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:56:21,036 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:56:21,036 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:56:21,036 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:56:21,036 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:56:21,037 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:21,037 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:21,038 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:56:21,038 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:21,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:56:21,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:56:21,038 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:21,038 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:56:21,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:21,039 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:21,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:56:21,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:21,039 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:56:21,039 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:56:21,040 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:56:21,040 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:56:21,040 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:56:21,040 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:56:21,040 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:56:21,040 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:56:21,040 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:21,040 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. started... 2023-04-19 20:56:21,040 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6873ba93789e63dd9f14391208147267 3/3 column families, dataSize=134.18 KB heapSize=384.14 KB 2023-04-19 20:56:21,040 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6873ba93789e63dd9f14391208147267, store=A 2023-04-19 20:56:21,040 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 20:56:21,041 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6873ba93789e63dd9f14391208147267, store=B 2023-04-19 20:56:21,041 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 20:56:21,041 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6873ba93789e63dd9f14391208147267, store=C 2023-04-19 20:56:21,041 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 20:56:21,056 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230419284fa4e20d244846ba2d5a60871ac3cc to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419284fa4e20d244846ba2d5a60871ac3cc 2023-04-19 20:56:21,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(9158): Flush requested on 6873ba93789e63dd9f14391208147267 2023-04-19 20:56:21,202 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. as already flushing 2023-04-19 20:56:21,209 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:21,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57236 deadline: 1681937841205, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:21,210 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:21,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57210 deadline: 1681937841206, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:21,213 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:21,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57204 deadline: 1681937841208, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:21,214 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:21,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57246 deadline: 1681937841209, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:21,214 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:21,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57226 deadline: 1681937841209, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:21,313 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:21,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57210 deadline: 1681937841311, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:21,317 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:21,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57204 deadline: 1681937841314, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:21,317 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:21,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57246 deadline: 1681937841315, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:21,317 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:21,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57226 deadline: 1681937841315, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:21,393 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/B/eee3503502504cb08b4a03de1760cd79 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/eee3503502504cb08b4a03de1760cd79 2023-04-19 20:56:21,399 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 6873ba93789e63dd9f14391208147267/B of 6873ba93789e63dd9f14391208147267 into eee3503502504cb08b4a03de1760cd79(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 20:56:21,399 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6873ba93789e63dd9f14391208147267: 2023-04-19 20:56:21,399 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267., storeName=6873ba93789e63dd9f14391208147267/B, priority=11, startTime=1681937780964; duration=0sec 2023-04-19 20:56:21,399 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 20:56:21,399 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-19 20:56:21,401 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61680 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-19 20:56:21,401 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1912): 6873ba93789e63dd9f14391208147267/C is initiating minor compaction (all files) 2023-04-19 20:56:21,401 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6873ba93789e63dd9f14391208147267/C in TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:21,401 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/826a0a4407cb488db170becc2217b51c, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/e3c72c0402434b0d999f0f6483a108d9, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/4f6b7a4ac90844378013533328745a19, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/11f608a8f60f486a8094a7304764ae55, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/0f6ae49ecfe74a13bb4bc05706a01e7d] into tmpdir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp, totalSize=60.2 K 2023-04-19 20:56:21,402 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 826a0a4407cb488db170becc2217b51c, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=325, earliestPutTs=1681937777708 2023-04-19 20:56:21,403 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting e3c72c0402434b0d999f0f6483a108d9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=363, earliestPutTs=1681937777795 2023-04-19 20:56:21,403 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 4f6b7a4ac90844378013533328745a19, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=377, earliestPutTs=1681937778419 2023-04-19 20:56:21,404 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 11f608a8f60f486a8094a7304764ae55, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=413, earliestPutTs=1681937779358 2023-04-19 20:56:21,405 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 0f6ae49ecfe74a13bb4bc05706a01e7d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=424, earliestPutTs=1681937780486 2023-04-19 20:56:21,407 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/A/90e84052bff54e508728bfe237e575ed as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/90e84052bff54e508728bfe237e575ed 2023-04-19 20:56:21,412 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 6873ba93789e63dd9f14391208147267/A of 6873ba93789e63dd9f14391208147267 into 90e84052bff54e508728bfe237e575ed(size=26.4 K), total size for store is 26.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 20:56:21,413 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6873ba93789e63dd9f14391208147267: 2023-04-19 20:56:21,413 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267., storeName=6873ba93789e63dd9f14391208147267/A, priority=11, startTime=1681937780964; duration=0sec 2023-04-19 20:56:21,413 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 20:56:21,414 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] throttle.PressureAwareThroughputController(145): 6873ba93789e63dd9f14391208147267#C#compaction#629 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 20:56:21,422 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/C/fa95bf53b19645f29d784ed3d3a5632e as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/fa95bf53b19645f29d784ed3d3a5632e 2023-04-19 20:56:21,426 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 6873ba93789e63dd9f14391208147267/C of 6873ba93789e63dd9f14391208147267 into fa95bf53b19645f29d784ed3d3a5632e(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 20:56:21,426 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6873ba93789e63dd9f14391208147267: 2023-04-19 20:56:21,426 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267., storeName=6873ba93789e63dd9f14391208147267/C, priority=11, startTime=1681937780964; duration=0sec 2023-04-19 20:56:21,426 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 20:56:21,466 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=478, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/A/37ab81a43c16427e877a21033f390d02 2023-04-19 20:56:21,476 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=478 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/B/bc443c74b06c4928986c3bf33c945127 2023-04-19 20:56:21,497 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=478 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/C/11379df5f405422ab142bf9546f8863d 2023-04-19 20:56:21,502 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/A/37ab81a43c16427e877a21033f390d02 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/37ab81a43c16427e877a21033f390d02 2023-04-19 20:56:21,506 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/37ab81a43c16427e877a21033f390d02, entries=150, sequenceid=478, filesize=25.4 K 2023-04-19 20:56:21,507 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/B/bc443c74b06c4928986c3bf33c945127 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/bc443c74b06c4928986c3bf33c945127 2023-04-19 20:56:21,512 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/bc443c74b06c4928986c3bf33c945127, entries=150, sequenceid=478, filesize=11.9 K 2023-04-19 20:56:21,513 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/C/11379df5f405422ab142bf9546f8863d as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/11379df5f405422ab142bf9546f8863d 2023-04-19 20:56:21,515 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:21,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57210 deadline: 1681937841514, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:21,517 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/11379df5f405422ab142bf9546f8863d, entries=150, sequenceid=478, filesize=11.9 K 2023-04-19 20:56:21,518 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.09 KB/393312, currentSize=53.67 KB/54960 for 6873ba93789e63dd9f14391208147267 in 478ms, sequenceid=478, compaction requested=false 2023-04-19 20:56:21,518 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6873ba93789e63dd9f14391208147267: 2023-04-19 20:56:21,518 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:21,518 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:56:21,518 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:56:21,518 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:21,518 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:56:21,518 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:56:21,521 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:21,521 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:21,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:21,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:21,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:21,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(9158): Flush requested on 6873ba93789e63dd9f14391208147267 2023-04-19 20:56:21,521 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 6873ba93789e63dd9f14391208147267 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-19 20:56:21,521 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:21,521 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:56:21,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:21,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:21,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:21,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:21,522 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6873ba93789e63dd9f14391208147267, store=A 2023-04-19 20:56:21,522 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:56:21,522 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6873ba93789e63dd9f14391208147267, store=B 2023-04-19 20:56:21,522 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:56:21,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:21,522 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6873ba93789e63dd9f14391208147267, store=C 2023-04-19 20:56:21,522 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:56:21,523 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:56:21,523 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2e5b365b[Count = 0] remaining members to acquire global barrier 2023-04-19 20:56:21,523 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:56:21,523 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:21,524 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:21,524 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:21,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:21,524 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:21,524 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:56:21,529 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:56:21,529 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:56:21,529 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:56:21,531 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:21,531 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:56:21,531 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:56:21,531 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:56:21,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:21,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:21,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:21,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:21,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:21,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:21,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:21,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:21,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:21,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:21,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:56:21,534 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:56:21,534 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:56:21,535 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:56:21,535 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:56:21,535 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:56:21,535 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:56:21,538 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:21,538 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:21,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:21,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:21,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:21,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:21,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:21,538 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:56:21,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:21,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:56:21,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:56:21,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:21,540 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:21,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:21,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:21,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:21,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:21,541 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:21,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:21,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:21,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:21,545 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:21,545 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:21,545 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:56:21,546 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:56:21,545 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:56:21,545 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:21,546 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:21,546 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:56:21,546 DEBUG [Thread-4043] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:56:21,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:56:21,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:56:21,546 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:56:21,546 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:21,546 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:21,546 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:21,546 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230419a45448a74b0541fa9565250d0b297c33 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419a45448a74b0541fa9565250d0b297c33 2023-04-19 20:56:21,546 DEBUG [Thread-4043] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:56:21,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:56:21,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:56:21,547 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:21,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57204 deadline: 1681937841544, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:21,548 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:21,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57226 deadline: 1681937841545, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:21,550 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:21,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57246 deadline: 1681937841548, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:21,646 DEBUG [Thread-4043] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:56:21,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:56:21,648 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:56:21,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:56:21,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:56:21,652 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:21,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57204 deadline: 1681937841650, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:21,653 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:56:21,653 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:56:21,653 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:56:21,653 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:56:21,654 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:21,654 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:21,655 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:21,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57226 deadline: 1681937841653, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:21,655 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:21,655 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:21,655 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:56:21,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57246 deadline: 1681937841654, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:21,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:56:21,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:56:21,656 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:21,656 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:56:21,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:21,656 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:21,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:56:21,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:21,657 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:56:21,673 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:56:21,673 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:56:21,673 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:56:21,673 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:56:21,673 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:56:21,673 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:56:21,673 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:21,673 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. started... 2023-04-19 20:56:21,673 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:56:21,674 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. as already flushing 2023-04-19 20:56:21,674 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:21,674 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:56:21,674 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:56:21,674 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:21,674 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:56:21,674 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:56:21,675 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:21,675 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:21,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:21,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:21,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:21,676 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:21,676 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:56:21,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:21,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:21,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:21,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:21,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:21,677 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:56:21,677 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1f76d899[Count = 0] remaining members to acquire global barrier 2023-04-19 20:56:21,677 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:56:21,677 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:21,678 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:21,678 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:21,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:21,678 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:21,678 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:56:21,678 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:56:21,679 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:56:21,679 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:56:21,680 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:21,680 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:56:21,680 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:56:21,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:21,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:21,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:21,681 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:56:21,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:21,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:21,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:21,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:21,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:21,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:21,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:21,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:56:21,683 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:56:21,683 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:56:21,683 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:56:21,683 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:56:21,683 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:56:21,683 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:56:21,685 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:21,685 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:56:21,685 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:21,685 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:21,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:21,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:21,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:21,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:21,685 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:56:21,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:56:21,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:21,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:21,687 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:21,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:21,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:21,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:21,688 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:21,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:21,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:21,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:21,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:21,690 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:21,690 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:21,690 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:21,690 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:21,690 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:21,691 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:21,691 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:21,691 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:56:21,691 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:56:21,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:56:21,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:56:21,691 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:56:21,691 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:56:21,691 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:56:21,691 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:56:21,691 DEBUG [Thread-4043] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:56:21,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:56:21,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:56:21,691 DEBUG [Thread-4043] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:56:21,714 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:21,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57236 deadline: 1681937841712, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:21,791 DEBUG [Thread-4043] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:56:21,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:56:21,793 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:56:21,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:56:21,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:56:21,795 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:56:21,795 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:56:21,795 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:56:21,795 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:56:21,795 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:21,795 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:21,797 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:21,797 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:56:21,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:56:21,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:56:21,797 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:21,797 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:56:21,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:21,798 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:21,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:56:21,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:21,798 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:56:21,798 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:56:21,798 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:56:21,798 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:56:21,798 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:56:21,798 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:56:21,799 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:56:21,799 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:56:21,799 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:21,799 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. started... 2023-04-19 20:56:21,799 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. as already flushing 2023-04-19 20:56:21,799 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:21,799 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:56:21,799 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:56:21,799 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:21,799 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:56:21,799 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:56:21,802 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:21,802 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:21,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:21,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:21,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:21,802 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:21,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:21,802 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:56:21,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:21,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:21,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:21,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:21,803 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:56:21,804 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@71b6a3f7[Count = 0] remaining members to acquire global barrier 2023-04-19 20:56:21,804 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:56:21,804 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:21,805 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:21,805 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:21,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:21,805 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:56:21,805 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:56:21,805 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:21,805 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:56:21,805 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:56:21,813 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:56:21,813 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:56:21,813 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:21,813 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:56:21,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:21,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:21,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:21,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:21,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:21,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:21,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:21,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:21,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:21,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:21,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:56:21,815 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:56:21,815 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:56:21,815 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:56:21,815 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:56:21,815 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:56:21,815 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:56:21,816 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:21,816 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:56:21,817 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:21,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:21,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:21,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:21,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:21,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:21,817 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:21,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:56:21,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:56:21,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:21,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:21,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:21,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:21,818 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:21,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:21,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:21,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:21,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:21,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:21,820 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:21,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57210 deadline: 1681937841819, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:21,820 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:21,820 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:21,820 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:21,820 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:56:21,820 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:56:21,820 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:56:21,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:56:21,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:56:21,820 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:21,821 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:21,821 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:21,821 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:21,820 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:56:21,820 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:56:21,821 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:56:21,821 DEBUG [Thread-4043] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:56:21,821 DEBUG [Thread-4043] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:56:21,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:56:21,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:56:21,855 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:21,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57204 deadline: 1681937841854, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:21,857 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:21,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57246 deadline: 1681937841856, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:21,861 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:21,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57226 deadline: 1681937841860, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:21,921 DEBUG [Thread-4043] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:56:21,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:56:21,923 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:56:21,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:56:21,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:56:21,924 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:56:21,924 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:56:21,924 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:56:21,925 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:56:21,925 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:21,925 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:21,929 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:56:21,929 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:21,930 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:56:21,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:56:21,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:21,930 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:21,930 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:56:21,930 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:21,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:56:21,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:21,931 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:56:21,931 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:56:21,931 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:56:21,931 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:56:21,931 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:56:21,931 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:56:21,931 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:56:21,931 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:56:21,931 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:21,931 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. started... 2023-04-19 20:56:21,931 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. as already flushing 2023-04-19 20:56:21,931 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:21,932 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:56:21,932 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:56:21,932 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:21,932 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:56:21,932 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:56:21,933 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:21,933 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:21,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:21,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:21,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:21,933 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:21,933 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:56:21,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:21,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:21,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:21,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:21,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:21,935 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:56:21,935 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@59bf08c7[Count = 0] remaining members to acquire global barrier 2023-04-19 20:56:21,935 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:56:21,935 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:21,936 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:21,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:21,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:21,936 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:56:21,936 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:56:21,936 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:56:21,936 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:21,936 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:56:21,937 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:56:21,937 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:56:21,937 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:21,938 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:56:21,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:21,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:21,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:21,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:21,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:21,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:21,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:21,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:21,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:21,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:21,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:56:21,940 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:56:21,940 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:56:21,940 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:56:21,940 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:56:21,940 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:56:21,940 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:56:21,943 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:21,943 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:56:21,943 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:21,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:21,943 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:56:21,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:56:21,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:21,945 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:21,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:21,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:21,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:21,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:21,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:21,960 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:21,960 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=494, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/A/151c342063d7435c960dcd3d8eb1c2a5 2023-04-19 20:56:21,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:21,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:21,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:21,961 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:21,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:21,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:21,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:21,964 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:21,965 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:21,965 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:21,965 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:56:21,965 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:56:21,965 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:21,965 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:21,965 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:56:21,965 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:21,965 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:56:21,965 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:56:21,965 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:56:21,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:56:21,965 DEBUG [Thread-4043] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:56:21,965 DEBUG [Thread-4043] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:56:21,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:56:21,965 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:21,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:56:21,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:56:21,971 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=494 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/B/3b64653e9af54c37a7f3ac49f59f3c54 2023-04-19 20:56:21,980 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=494 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/C/ea89bf0f7e3c4e87857c7bb120c24c77 2023-04-19 20:56:21,985 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/A/151c342063d7435c960dcd3d8eb1c2a5 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/151c342063d7435c960dcd3d8eb1c2a5 2023-04-19 20:56:21,988 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/151c342063d7435c960dcd3d8eb1c2a5, entries=200, sequenceid=494, filesize=32.3 K 2023-04-19 20:56:21,988 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/B/3b64653e9af54c37a7f3ac49f59f3c54 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/3b64653e9af54c37a7f3ac49f59f3c54 2023-04-19 20:56:21,992 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/3b64653e9af54c37a7f3ac49f59f3c54, entries=150, sequenceid=494, filesize=11.9 K 2023-04-19 20:56:21,992 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/C/ea89bf0f7e3c4e87857c7bb120c24c77 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/ea89bf0f7e3c4e87857c7bb120c24c77 2023-04-19 20:56:21,995 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/ea89bf0f7e3c4e87857c7bb120c24c77, entries=150, sequenceid=494, filesize=11.9 K 2023-04-19 20:56:21,996 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 6873ba93789e63dd9f14391208147267 in 475ms, sequenceid=494, compaction requested=true 2023-04-19 20:56:21,996 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 6873ba93789e63dd9f14391208147267: 2023-04-19 20:56:21,996 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 20:56:21,996 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 20:56:21,996 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-19 20:56:21,996 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-19 20:56:21,996 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-19 20:56:21,997 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 86132 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-19 20:56:21,997 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37512 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-19 20:56:21,997 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1912): 6873ba93789e63dd9f14391208147267/A is initiating minor compaction (all files) 2023-04-19 20:56:21,997 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1912): 6873ba93789e63dd9f14391208147267/B is initiating minor compaction (all files) 2023-04-19 20:56:21,997 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 6873ba93789e63dd9f14391208147267/A in TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:21,997 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6873ba93789e63dd9f14391208147267/B in TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:21,997 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/90e84052bff54e508728bfe237e575ed, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/37ab81a43c16427e877a21033f390d02, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/151c342063d7435c960dcd3d8eb1c2a5] into tmpdir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp, totalSize=84.1 K 2023-04-19 20:56:21,997 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/eee3503502504cb08b4a03de1760cd79, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/bc443c74b06c4928986c3bf33c945127, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/3b64653e9af54c37a7f3ac49f59f3c54] into tmpdir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp, totalSize=36.6 K 2023-04-19 20:56:21,997 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 90e84052bff54e508728bfe237e575ed, keycount=150, bloomtype=ROW, size=26.4 K, encoding=NONE, compression=NONE, seqNum=424, earliestPutTs=1681937780486 2023-04-19 20:56:21,997 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting eee3503502504cb08b4a03de1760cd79, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=424, earliestPutTs=1681937780486 2023-04-19 20:56:21,998 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 37ab81a43c16427e877a21033f390d02, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=478, earliestPutTs=1681937780584 2023-04-19 20:56:21,998 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting bc443c74b06c4928986c3bf33c945127, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=478, earliestPutTs=1681937780584 2023-04-19 20:56:21,998 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 151c342063d7435c960dcd3d8eb1c2a5, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=494, earliestPutTs=1681937781208 2023-04-19 20:56:21,998 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 3b64653e9af54c37a7f3ac49f59f3c54, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=494, earliestPutTs=1681937781208 2023-04-19 20:56:22,003 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] throttle.PressureAwareThroughputController(145): 6873ba93789e63dd9f14391208147267#B#compaction#635 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 20:56:22,010 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] throttle.PressureAwareThroughputController(145): 6873ba93789e63dd9f14391208147267#A#compaction#636 average throughput is 3.29 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 20:56:22,015 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/B/f48c5241a55842d6812e8404b9e4c5b2 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/f48c5241a55842d6812e8404b9e4c5b2 2023-04-19 20:56:22,020 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 6873ba93789e63dd9f14391208147267/B of 6873ba93789e63dd9f14391208147267 into f48c5241a55842d6812e8404b9e4c5b2(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 20:56:22,020 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6873ba93789e63dd9f14391208147267: 2023-04-19 20:56:22,020 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267., storeName=6873ba93789e63dd9f14391208147267/B, priority=13, startTime=1681937781996; duration=0sec 2023-04-19 20:56:22,020 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 20:56:22,020 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-19 20:56:22,021 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37478 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-19 20:56:22,021 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1912): 6873ba93789e63dd9f14391208147267/C is initiating minor compaction (all files) 2023-04-19 20:56:22,021 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6873ba93789e63dd9f14391208147267/C in TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:22,021 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/fa95bf53b19645f29d784ed3d3a5632e, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/11379df5f405422ab142bf9546f8863d, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/ea89bf0f7e3c4e87857c7bb120c24c77] into tmpdir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp, totalSize=36.6 K 2023-04-19 20:56:22,022 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting fa95bf53b19645f29d784ed3d3a5632e, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=424, earliestPutTs=1681937780486 2023-04-19 20:56:22,022 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 11379df5f405422ab142bf9546f8863d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=478, earliestPutTs=1681937780584 2023-04-19 20:56:22,022 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting ea89bf0f7e3c4e87857c7bb120c24c77, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=494, earliestPutTs=1681937781208 2023-04-19 20:56:22,028 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] throttle.PressureAwareThroughputController(145): 6873ba93789e63dd9f14391208147267#C#compaction#637 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 20:56:22,049 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/C/3b0a0c2531804a32bb02046673d6ba7e as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/3b0a0c2531804a32bb02046673d6ba7e 2023-04-19 20:56:22,053 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 6873ba93789e63dd9f14391208147267/C of 6873ba93789e63dd9f14391208147267 into 3b0a0c2531804a32bb02046673d6ba7e(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 20:56:22,054 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6873ba93789e63dd9f14391208147267: 2023-04-19 20:56:22,054 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267., storeName=6873ba93789e63dd9f14391208147267/C, priority=13, startTime=1681937781996; duration=0sec 2023-04-19 20:56:22,054 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 20:56:22,065 DEBUG [Thread-4043] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:56:22,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:56:22,068 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:56:22,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:56:22,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:56:22,069 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:56:22,069 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:56:22,069 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:56:22,069 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:56:22,069 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:22,070 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:22,071 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:56:22,071 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:22,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:56:22,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:56:22,071 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:22,071 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:56:22,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:22,072 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:22,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:56:22,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:22,072 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:56:22,072 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:56:22,072 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:56:22,072 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:56:22,072 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:56:22,073 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:56:22,073 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:56:22,073 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:22,073 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. started... 2023-04-19 20:56:22,073 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6873ba93789e63dd9f14391208147267 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-19 20:56:22,073 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6873ba93789e63dd9f14391208147267, store=A 2023-04-19 20:56:22,074 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:56:22,074 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6873ba93789e63dd9f14391208147267, store=B 2023-04-19 20:56:22,074 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:56:22,074 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6873ba93789e63dd9f14391208147267, store=C 2023-04-19 20:56:22,074 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:56:22,086 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304193ab4a47a6d8740719eff0f5fb9ec78ff to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304193ab4a47a6d8740719eff0f5fb9ec78ff 2023-04-19 20:56:22,092 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=515, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/A/d04fedd90da34bb2a8a3d5a76415b76f 2023-04-19 20:56:22,106 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=515 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/B/12386e413b51429f85b6b262ee1b5e82 2023-04-19 20:56:22,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(9158): Flush requested on 6873ba93789e63dd9f14391208147267 2023-04-19 20:56:22,160 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. as already flushing 2023-04-19 20:56:22,171 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:22,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57204 deadline: 1681937842168, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:22,171 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:22,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57246 deadline: 1681937842168, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:22,173 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:22,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57226 deadline: 1681937842171, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:22,273 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:22,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57204 deadline: 1681937842272, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:22,273 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:22,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57246 deadline: 1681937842272, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:22,276 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:22,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57226 deadline: 1681937842274, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:22,326 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:22,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57210 deadline: 1681937842325, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:22,457 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/A/0c87feb59d8e4991ae7f1290e16b1c96 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/0c87feb59d8e4991ae7f1290e16b1c96 2023-04-19 20:56:22,461 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 6873ba93789e63dd9f14391208147267/A of 6873ba93789e63dd9f14391208147267 into 0c87feb59d8e4991ae7f1290e16b1c96(size=26.5 K), total size for store is 26.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 20:56:22,461 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6873ba93789e63dd9f14391208147267: 2023-04-19 20:56:22,461 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267., storeName=6873ba93789e63dd9f14391208147267/A, priority=13, startTime=1681937781996; duration=0sec 2023-04-19 20:56:22,461 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 20:56:22,475 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:22,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57204 deadline: 1681937842474, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:22,475 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:22,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57246 deadline: 1681937842474, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:22,479 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:22,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57226 deadline: 1681937842479, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:22,516 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=515 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/C/aa108859b8444a58bc53bfe600264938 2023-04-19 20:56:22,520 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/A/d04fedd90da34bb2a8a3d5a76415b76f as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/d04fedd90da34bb2a8a3d5a76415b76f 2023-04-19 20:56:22,523 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/d04fedd90da34bb2a8a3d5a76415b76f, entries=150, sequenceid=515, filesize=25.4 K 2023-04-19 20:56:22,524 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/B/12386e413b51429f85b6b262ee1b5e82 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/12386e413b51429f85b6b262ee1b5e82 2023-04-19 20:56:22,527 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/12386e413b51429f85b6b262ee1b5e82, entries=150, sequenceid=515, filesize=11.9 K 2023-04-19 20:56:22,528 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/C/aa108859b8444a58bc53bfe600264938 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/aa108859b8444a58bc53bfe600264938 2023-04-19 20:56:22,531 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/aa108859b8444a58bc53bfe600264938, entries=150, sequenceid=515, filesize=11.9 K 2023-04-19 20:56:22,532 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 6873ba93789e63dd9f14391208147267 in 459ms, sequenceid=515, compaction requested=false 2023-04-19 20:56:22,532 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6873ba93789e63dd9f14391208147267: 2023-04-19 20:56:22,532 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:22,532 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:56:22,532 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:56:22,532 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:22,532 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:56:22,532 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:56:22,534 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:22,534 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:22,534 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:56:22,534 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:22,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:22,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:22,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:22,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:22,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:22,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:22,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:22,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:22,536 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:56:22,536 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2ea10445[Count = 0] remaining members to acquire global barrier 2023-04-19 20:56:22,536 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:56:22,536 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:22,542 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:22,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:22,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:22,542 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:56:22,542 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:56:22,542 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:56:22,543 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:22,543 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:56:22,544 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:56:22,544 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:56:22,544 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:22,544 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:56:22,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:22,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:22,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:22,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:22,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:22,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:22,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:22,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:22,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:22,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:22,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:56:22,546 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:56:22,546 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:56:22,546 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:56:22,546 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:56:22,546 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:56:22,546 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:56:22,548 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:22,548 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:56:22,548 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:22,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:22,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:22,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:22,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:22,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:22,549 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:22,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:56:22,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:56:22,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:22,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:22,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:22,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:22,550 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:22,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:22,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:22,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:22,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:22,552 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:22,552 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:22,552 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:22,552 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:56:22,552 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:56:22,552 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:56:22,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:56:22,553 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:22,553 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:22,553 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:22,553 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:22,553 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:56:22,553 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:56:22,553 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:56:22,553 DEBUG [Thread-4043] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:56:22,553 DEBUG [Thread-4043] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:56:22,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:56:22,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:56:22,553 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:56:22,653 DEBUG [Thread-4043] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:56:22,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:56:22,655 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:56:22,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:56:22,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:56:22,657 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:56:22,657 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:56:22,657 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:56:22,657 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:56:22,657 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:22,657 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:22,665 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:56:22,665 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:22,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:56:22,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:56:22,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:22,665 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:22,665 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:56:22,666 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:22,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:56:22,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:22,666 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:56:22,666 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:56:22,666 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:56:22,666 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:56:22,666 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:56:22,666 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:56:22,667 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:56:22,667 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:56:22,667 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:22,667 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. started... 2023-04-19 20:56:22,667 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6873ba93789e63dd9f14391208147267 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-19 20:56:22,667 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6873ba93789e63dd9f14391208147267, store=A 2023-04-19 20:56:22,667 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:56:22,667 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6873ba93789e63dd9f14391208147267, store=B 2023-04-19 20:56:22,667 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:56:22,667 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6873ba93789e63dd9f14391208147267, store=C 2023-04-19 20:56:22,667 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:56:22,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(9158): Flush requested on 6873ba93789e63dd9f14391208147267 2023-04-19 20:56:22,727 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. as already flushing 2023-04-19 20:56:22,777 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:22,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57246 deadline: 1681937842776, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:22,778 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:22,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57204 deadline: 1681937842776, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:22,780 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:22,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57236 deadline: 1681937842777, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:22,784 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:22,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57226 deadline: 1681937842782, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:22,882 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:22,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57236 deadline: 1681937842881, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:23,081 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230419829a92fb9a124004b0d08db3a66f12c6 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419829a92fb9a124004b0d08db3a66f12c6 2023-04-19 20:56:23,085 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:23,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57236 deadline: 1681937843083, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:23,086 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=529, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/A/60d5476aad5f4856a5ec1f39d2188a1e 2023-04-19 20:56:23,095 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=529 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/B/9efea23d519e445a9b3a894bdb478948 2023-04-19 20:56:23,283 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:23,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57246 deadline: 1681937843282, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:23,284 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:23,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57204 deadline: 1681937843282, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:23,288 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:23,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57226 deadline: 1681937843287, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:23,332 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:23,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57210 deadline: 1681937843330, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:23,390 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:23,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57236 deadline: 1681937843388, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:23,504 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=529 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/C/f1a7cebb8d78421081faf72f3fef352e 2023-04-19 20:56:23,507 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/A/60d5476aad5f4856a5ec1f39d2188a1e as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/60d5476aad5f4856a5ec1f39d2188a1e 2023-04-19 20:56:23,510 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/60d5476aad5f4856a5ec1f39d2188a1e, entries=150, sequenceid=529, filesize=25.4 K 2023-04-19 20:56:23,511 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/B/9efea23d519e445a9b3a894bdb478948 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/9efea23d519e445a9b3a894bdb478948 2023-04-19 20:56:23,514 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/9efea23d519e445a9b3a894bdb478948, entries=150, sequenceid=529, filesize=11.9 K 2023-04-19 20:56:23,515 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/C/f1a7cebb8d78421081faf72f3fef352e as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/f1a7cebb8d78421081faf72f3fef352e 2023-04-19 20:56:23,518 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/f1a7cebb8d78421081faf72f3fef352e, entries=150, sequenceid=529, filesize=11.9 K 2023-04-19 20:56:23,519 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 6873ba93789e63dd9f14391208147267 in 852ms, sequenceid=529, compaction requested=true 2023-04-19 20:56:23,519 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6873ba93789e63dd9f14391208147267: 2023-04-19 20:56:23,519 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:23,519 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:56:23,519 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:56:23,519 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:23,520 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:56:23,520 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:56:23,521 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:23,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:23,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:23,521 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:23,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:23,522 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:23,522 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:56:23,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:23,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:23,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:23,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:23,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:23,524 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:56:23,524 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7bab318[Count = 0] remaining members to acquire global barrier 2023-04-19 20:56:23,525 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:56:23,525 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:23,525 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:23,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:23,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:23,526 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:56:23,526 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:56:23,526 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:56:23,526 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:23,526 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:56:23,527 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:56:23,527 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:56:23,527 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:56:23,527 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:23,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:23,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:23,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:23,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:23,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:23,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:23,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:23,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:23,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:23,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:23,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:56:23,530 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:56:23,530 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:56:23,530 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:56:23,530 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:56:23,530 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:56:23,530 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:56:23,533 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:23,533 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:56:23,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:23,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:23,533 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:23,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:23,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:23,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:23,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:23,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:56:23,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:56:23,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:23,534 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:23,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:23,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:23,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:23,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:23,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:23,536 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:23,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:23,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:23,541 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:56:23,541 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:56:23,541 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:23,541 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:23,541 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:23,541 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:23,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:56:23,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:56:23,541 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:56:23,541 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:23,541 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:23,541 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:23,541 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:56:23,541 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:56:23,542 DEBUG [Thread-4043] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:56:23,542 DEBUG [Thread-4043] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:56:23,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:56:23,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:56:23,642 DEBUG [Thread-4043] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:56:23,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:56:23,644 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:56:23,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:56:23,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:56:23,645 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:56:23,645 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:56:23,645 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:56:23,645 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:56:23,646 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:23,646 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:23,647 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:23,647 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:56:23,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:56:23,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:56:23,647 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:23,647 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:56:23,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:23,647 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:23,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:56:23,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:23,648 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:56:23,648 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:56:23,648 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:56:23,648 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:56:23,648 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:56:23,648 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:56:23,649 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:56:23,649 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:56:23,649 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:23,649 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. started... 2023-04-19 20:56:23,649 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6873ba93789e63dd9f14391208147267 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-19 20:56:23,649 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6873ba93789e63dd9f14391208147267, store=A 2023-04-19 20:56:23,649 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:56:23,649 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6873ba93789e63dd9f14391208147267, store=B 2023-04-19 20:56:23,649 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:56:23,649 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6873ba93789e63dd9f14391208147267, store=C 2023-04-19 20:56:23,649 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:56:23,669 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304198fd3f56c29ac4920b120d85396963baf to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304198fd3f56c29ac4920b120d85396963baf 2023-04-19 20:56:23,681 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=548, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/A/ed7f3b4bb0d84c5cabb2c08befee59fc 2023-04-19 20:56:23,693 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=548 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/B/786395e45605458fb4c6447e2220cdd2 2023-04-19 20:56:23,703 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=548 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/C/2b4460932e38440099225f3c7589f7c1 2023-04-19 20:56:23,706 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/A/ed7f3b4bb0d84c5cabb2c08befee59fc as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/ed7f3b4bb0d84c5cabb2c08befee59fc 2023-04-19 20:56:23,710 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/ed7f3b4bb0d84c5cabb2c08befee59fc, entries=150, sequenceid=548, filesize=25.4 K 2023-04-19 20:56:23,711 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/B/786395e45605458fb4c6447e2220cdd2 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/786395e45605458fb4c6447e2220cdd2 2023-04-19 20:56:23,714 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/786395e45605458fb4c6447e2220cdd2, entries=150, sequenceid=548, filesize=11.9 K 2023-04-19 20:56:23,715 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/C/2b4460932e38440099225f3c7589f7c1 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/2b4460932e38440099225f3c7589f7c1 2023-04-19 20:56:23,718 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/2b4460932e38440099225f3c7589f7c1, entries=150, sequenceid=548, filesize=11.9 K 2023-04-19 20:56:23,719 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=0 B/0 for 6873ba93789e63dd9f14391208147267 in 70ms, sequenceid=548, compaction requested=true 2023-04-19 20:56:23,719 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6873ba93789e63dd9f14391208147267: 2023-04-19 20:56:23,719 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:23,719 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:56:23,719 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:56:23,719 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:23,719 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:56:23,719 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:56:23,722 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:23,722 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:23,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:23,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:23,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:23,722 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:23,723 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:56:23,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:23,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:23,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:23,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:23,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:23,724 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:56:23,724 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@43d70103[Count = 0] remaining members to acquire global barrier 2023-04-19 20:56:23,724 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:56:23,724 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:23,725 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:23,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:23,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:23,725 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:56:23,725 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:56:23,725 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:56:23,726 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:23,726 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:56:23,727 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:23,727 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:56:23,727 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:56:23,727 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:56:23,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:23,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:23,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:23,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:23,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:23,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:23,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:23,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:23,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:23,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:23,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:56:23,730 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:56:23,730 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:56:23,730 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:56:23,730 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:56:23,730 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:56:23,730 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:56:23,731 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:23,731 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:23,731 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:56:23,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:23,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:23,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:23,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:23,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:23,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:23,732 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:23,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:56:23,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:56:23,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:23,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:23,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:23,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:23,734 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:23,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:23,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:23,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:23,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:23,737 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:56:23,737 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:23,737 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:56:23,737 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:56:23,737 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:23,737 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:56:23,737 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:56:23,737 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:56:23,737 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:23,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:56:23,737 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:23,737 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:23,737 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:56:23,737 DEBUG [Thread-4043] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:56:23,737 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:23,738 DEBUG [Thread-4043] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:56:23,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:56:23,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:56:23,738 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:23,838 DEBUG [Thread-4043] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:56:23,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:56:23,840 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:56:23,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:56:23,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:56:23,841 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:56:23,841 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:56:23,841 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:56:23,841 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:56:23,842 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:23,842 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:23,843 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:23,843 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:56:23,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:56:23,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:56:23,843 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:23,843 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:56:23,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:23,844 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:23,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:56:23,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:23,844 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:56:23,845 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:56:23,845 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:56:23,845 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:56:23,845 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:56:23,845 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:56:23,845 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:56:23,845 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:56:23,845 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:23,845 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. started... 2023-04-19 20:56:23,846 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6873ba93789e63dd9f14391208147267: 2023-04-19 20:56:23,846 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:23,846 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:56:23,846 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:56:23,846 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:23,846 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:56:23,846 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:56:23,847 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:23,847 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:23,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:23,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:23,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:23,848 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:23,848 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:56:23,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:23,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:23,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:23,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:23,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:23,849 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:56:23,849 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@628fd08e[Count = 0] remaining members to acquire global barrier 2023-04-19 20:56:23,849 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:56:23,849 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:23,850 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:23,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:23,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:23,850 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:56:23,850 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:56:23,850 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:56:23,851 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:23,851 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:56:23,852 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:23,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:23,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:23,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:23,852 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:56:23,853 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:56:23,853 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:56:23,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:23,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:23,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:23,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:23,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:23,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:23,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:23,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:56:23,855 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:56:23,855 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:56:23,855 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:56:23,855 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:56:23,855 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:56:23,855 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:56:23,856 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:23,856 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:23,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:23,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:23,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:23,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:23,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:23,857 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:56:23,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:56:23,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:56:23,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:23,857 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:23,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:23,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:23,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:23,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:23,858 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:23,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:23,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:23,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:23,861 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:23,861 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:23,861 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:56:23,861 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:56:23,861 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:56:23,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:56:23,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:56:23,861 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:56:23,861 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:56:23,861 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:23,861 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:23,861 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:23,861 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:23,861 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:23,861 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:56:23,861 DEBUG [Thread-4043] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:56:23,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:56:23,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:56:23,861 DEBUG [Thread-4043] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:56:23,965 DEBUG [Thread-4043] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:56:23,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:56:23,967 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:56:23,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:56:23,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:56:23,976 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:56:23,976 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:56:23,976 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:56:23,976 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:56:23,977 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:23,977 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:23,978 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:56:23,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:56:23,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:56:23,978 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:23,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:23,978 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:23,978 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:56:23,979 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:23,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:56:23,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:23,979 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:56:23,979 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:56:23,979 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:56:23,979 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:56:23,979 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:56:23,979 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:56:23,979 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:56:23,980 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:56:23,980 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:23,980 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. started... 2023-04-19 20:56:23,980 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6873ba93789e63dd9f14391208147267 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-19 20:56:23,994 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6873ba93789e63dd9f14391208147267, store=A 2023-04-19 20:56:23,994 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:56:23,994 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6873ba93789e63dd9f14391208147267, store=B 2023-04-19 20:56:23,994 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:56:23,994 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6873ba93789e63dd9f14391208147267, store=C 2023-04-19 20:56:23,994 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:56:24,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(9158): Flush requested on 6873ba93789e63dd9f14391208147267 2023-04-19 20:56:24,003 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. as already flushing 2023-04-19 20:56:24,007 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304194b72c7d0b4224bb48b607d3bdad86621 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304194b72c7d0b4224bb48b607d3bdad86621 2023-04-19 20:56:24,029 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=558, memsize=13.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/A/aee3852a71d64a3f9b9771d3f537012c 2023-04-19 20:56:24,069 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681937784050, max timestamp=1681937784068Num uniques 859; 2023-04-19 20:56:24,069 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 20:56:24,070 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681937784050, max timestamp=1681937784068Num uniques 859; 2023-04-19 20:56:24,070 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 20:56:24,070 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681937784050, max timestamp=1681937784068Num uniques 859; 2023-04-19 20:56:24,070 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 20:56:24,134 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681937784101, max timestamp=1681937784132Num uniques 1018; 2023-04-19 20:56:24,135 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681937784101, max timestamp=1681937784132Num uniques 1018; 2023-04-19 20:56:24,135 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681937784101, max timestamp=1681937784132Num uniques 1018; 2023-04-19 20:56:24,135 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-19 20:56:24,135 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-19 20:56:24,135 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-19 20:56:24,202 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1681937784183, max timestamp=1681937784199Num uniques 1027; 2023-04-19 20:56:24,202 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1681937784183, max timestamp=1681937784199Num uniques 1027; 2023-04-19 20:56:24,202 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-19 20:56:24,202 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1681937784183, max timestamp=1681937784199Num uniques 1027; 2023-04-19 20:56:24,202 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-19 20:56:24,202 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-19 20:56:24,266 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1681937784248, max timestamp=1681937784264Num uniques 1036; 2023-04-19 20:56:24,266 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-04-19 20:56:24,267 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1681937784248, max timestamp=1681937784264Num uniques 1036; 2023-04-19 20:56:24,267 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-04-19 20:56:24,267 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1681937784248, max timestamp=1681937784264Num uniques 1036; 2023-04-19 20:56:24,267 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-04-19 20:56:24,302 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1681937784297, max timestamp=1681937784301Num uniques 1045; 2023-04-19 20:56:24,302 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-04-19 20:56:24,303 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1681937784297, max timestamp=1681937784301Num uniques 1045; 2023-04-19 20:56:24,304 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1681937784297, max timestamp=1681937784301Num uniques 1045; 2023-04-19 20:56:24,304 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-04-19 20:56:24,304 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-04-19 20:56:24,323 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:24,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57204 deadline: 1681937844320, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:24,324 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:24,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57226 deadline: 1681937844321, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:24,326 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:24,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 293 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57236 deadline: 1681937844322, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:24,326 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:24,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57246 deadline: 1681937844323, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:24,429 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:24,429 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:24,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57226 deadline: 1681937844425, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:24,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57204 deadline: 1681937844425, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:24,430 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:24,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 295 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57236 deadline: 1681937844427, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:24,430 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:24,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57246 deadline: 1681937844427, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:24,442 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=558 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/B/630dfd12107742b2a23a7044ee8e0d6e 2023-04-19 20:56:24,633 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:24,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 297 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57236 deadline: 1681937844631, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:24,633 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:24,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57226 deadline: 1681937844631, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:24,634 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:24,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57246 deadline: 1681937844631, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:24,634 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:24,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57204 deadline: 1681937844631, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:24,851 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=558 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/C/ae53b4179b724aa888e021e60fb3c784 2023-04-19 20:56:24,855 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/A/aee3852a71d64a3f9b9771d3f537012c as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/aee3852a71d64a3f9b9771d3f537012c 2023-04-19 20:56:24,860 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/aee3852a71d64a3f9b9771d3f537012c, entries=100, sequenceid=558, filesize=18.6 K 2023-04-19 20:56:24,861 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/B/630dfd12107742b2a23a7044ee8e0d6e as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/630dfd12107742b2a23a7044ee8e0d6e 2023-04-19 20:56:24,864 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/630dfd12107742b2a23a7044ee8e0d6e, entries=50, sequenceid=558, filesize=7.1 K 2023-04-19 20:56:24,864 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/C/ae53b4179b724aa888e021e60fb3c784 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/ae53b4179b724aa888e021e60fb3c784 2023-04-19 20:56:24,867 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/ae53b4179b724aa888e021e60fb3c784, entries=50, sequenceid=558, filesize=7.1 K 2023-04-19 20:56:24,868 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=140.89 KB/144270 for 6873ba93789e63dd9f14391208147267 in 888ms, sequenceid=558, compaction requested=true 2023-04-19 20:56:24,868 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6873ba93789e63dd9f14391208147267: 2023-04-19 20:56:24,868 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:24,868 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:56:24,868 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:56:24,868 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:24,868 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:56:24,868 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:56:24,871 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:24,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:24,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:24,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:24,871 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:24,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:24,872 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:24,872 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:56:24,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:24,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:24,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:24,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:24,874 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:56:24,874 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@44aa73b6[Count = 0] remaining members to acquire global barrier 2023-04-19 20:56:24,874 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:56:24,874 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:24,875 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:24,875 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:24,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:24,875 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:56:24,875 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:56:24,875 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:24,875 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:56:24,875 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:56:24,876 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:24,876 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:56:24,876 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:56:24,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:24,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:24,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:24,876 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:56:24,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:24,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:24,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:24,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:24,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:24,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:24,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:24,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:56:24,879 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:56:24,879 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:56:24,879 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:56:24,879 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:56:24,879 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:56:24,879 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:56:24,880 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:24,880 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:24,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:24,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:24,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:24,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:24,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:24,881 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:56:24,881 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:24,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:56:24,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:56:24,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:24,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:24,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:24,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:24,882 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:24,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:24,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:24,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:24,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:24,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:24,888 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:24,888 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:56:24,888 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:24,888 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:24,888 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:56:24,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:56:24,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:56:24,888 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:56:24,888 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:56:24,888 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:56:24,888 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:56:24,888 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:24,889 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:24,889 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:24,889 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:24,889 DEBUG [Thread-4043] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:56:24,889 DEBUG [Thread-4043] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:56:24,889 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:56:24,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:56:24,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(9158): Flush requested on 6873ba93789e63dd9f14391208147267 2023-04-19 20:56:24,938 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 6873ba93789e63dd9f14391208147267 3/3 column families, dataSize=147.60 KB heapSize=422.16 KB 2023-04-19 20:56:24,939 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1681937784319, max timestamp=1681937784937Num uniques 1054; 2023-04-19 20:56:24,939 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6873ba93789e63dd9f14391208147267, store=A 2023-04-19 20:56:24,939 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=7050, suffix heap size=137584,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=154 2023-04-19 20:56:24,939 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-19 20:56:24,939 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6873ba93789e63dd9f14391208147267, store=B 2023-04-19 20:56:24,940 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-19 20:56:24,940 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6873ba93789e63dd9f14391208147267, store=C 2023-04-19 20:56:24,940 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 20:56:24,953 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:24,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57226 deadline: 1681937844949, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:24,955 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:24,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 302 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57236 deadline: 1681937844951, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:24,955 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:24,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57246 deadline: 1681937844952, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:24,956 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:24,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57204 deadline: 1681937844953, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:24,989 DEBUG [Thread-4043] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:56:24,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:56:24,991 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:56:24,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:56:24,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:56:24,992 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:56:24,992 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:56:24,992 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:56:24,992 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:56:24,993 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:24,993 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:24,995 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:56:24,995 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:24,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:56:24,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:56:24,995 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:24,995 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:56:24,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:24,996 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:24,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:56:24,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:24,997 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:56:24,997 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:56:24,997 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:56:24,997 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:56:24,997 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:56:24,997 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:56:24,997 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:56:24,997 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:56:24,997 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:24,997 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. started... 2023-04-19 20:56:24,997 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. as already flushing 2023-04-19 20:56:24,998 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:24,998 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:56:24,998 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:56:24,998 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:24,998 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:56:24,998 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:56:24,999 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:24,999 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:24,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:24,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:24,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:24,999 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:24,999 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:56:25,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:25,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:25,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:25,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:25,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:25,001 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:56:25,001 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@165a3a48[Count = 0] remaining members to acquire global barrier 2023-04-19 20:56:25,001 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:56:25,001 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:25,002 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:25,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:25,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:25,002 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:56:25,002 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:56:25,002 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:56:25,002 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:25,002 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:56:25,005 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:25,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:25,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:25,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:25,005 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:56:25,005 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:56:25,005 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:56:25,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:25,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:25,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:25,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:25,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:25,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:25,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:25,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:56:25,009 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:56:25,009 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:56:25,009 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:56:25,009 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:56:25,009 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:56:25,009 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:56:25,010 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:25,010 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:56:25,010 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:25,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:25,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:25,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:25,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:25,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:25,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:56:25,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:56:25,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:25,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:25,012 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:25,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:25,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:25,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:25,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:25,013 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:25,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:25,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:25,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:25,015 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:56:25,015 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:56:25,015 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:56:25,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:56:25,016 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:25,016 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:25,016 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:25,016 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:56:25,016 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:56:25,016 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:56:25,016 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:25,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:56:25,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:56:25,016 DEBUG [Thread-4043] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:56:25,016 DEBUG [Thread-4043] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:56:25,016 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:56:25,016 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:25,016 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:25,016 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:25,055 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:25,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57226 deadline: 1681937845054, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:25,059 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:25,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 304 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57236 deadline: 1681937845056, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:25,059 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:25,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57246 deadline: 1681937845056, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:25,060 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:25,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57204 deadline: 1681937845057, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:25,116 DEBUG [Thread-4043] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:56:25,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:56:25,119 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:56:25,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:56:25,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:56:25,120 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:56:25,120 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:56:25,120 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:56:25,120 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:56:25,121 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:25,121 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:25,122 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:56:25,122 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:25,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:56:25,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:56:25,122 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:25,123 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:56:25,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:25,123 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:25,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:56:25,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:25,123 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:56:25,124 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:56:25,124 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:56:25,124 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:56:25,124 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:56:25,124 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:56:25,124 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:56:25,124 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:56:25,124 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:25,124 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. started... 2023-04-19 20:56:25,124 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. as already flushing 2023-04-19 20:56:25,124 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:25,125 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:56:25,125 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:56:25,125 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:25,125 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:56:25,125 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:56:25,126 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:25,126 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:25,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:25,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:25,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:25,127 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:25,127 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:56:25,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:25,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:25,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:25,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:25,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:25,129 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:56:25,129 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6dfefe99[Count = 0] remaining members to acquire global barrier 2023-04-19 20:56:25,129 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:56:25,129 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:25,130 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:25,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:25,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:25,130 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:56:25,130 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:56:25,130 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:56:25,130 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:25,130 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:56:25,133 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:56:25,133 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:56:25,133 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:56:25,133 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:25,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:25,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:25,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:25,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:25,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:25,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:25,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:25,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:25,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:25,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:25,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:56:25,136 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:56:25,136 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:56:25,136 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:56:25,136 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:56:25,136 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:56:25,136 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:56:25,138 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:25,138 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:56:25,138 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:25,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:25,138 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:25,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:25,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:25,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:25,139 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:56:25,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:56:25,139 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:25,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:25,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:25,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:25,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:25,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:25,140 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:25,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:25,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:25,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:25,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:25,142 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:25,142 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:25,142 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:56:25,142 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:25,142 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:56:25,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:56:25,142 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:56:25,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:56:25,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:56:25,143 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:56:25,143 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:56:25,143 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:56:25,143 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:56:25,143 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:25,143 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:25,143 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:25,143 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:25,143 DEBUG [Thread-4043] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:56:25,143 DEBUG [Thread-4043] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:56:25,243 DEBUG [Thread-4043] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:56:25,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:56:25,245 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:56:25,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:56:25,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:56:25,247 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:56:25,247 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:56:25,247 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:56:25,247 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:56:25,247 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:25,247 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:25,248 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:25,248 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:56:25,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:56:25,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:56:25,249 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:25,249 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:56:25,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:25,249 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:25,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:56:25,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:25,249 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:56:25,250 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:56:25,250 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:56:25,250 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:56:25,250 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:56:25,250 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:56:25,250 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:56:25,250 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:56:25,250 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:25,250 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. started... 2023-04-19 20:56:25,250 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. as already flushing 2023-04-19 20:56:25,250 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:25,250 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:56:25,250 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:56:25,250 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:25,250 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:56:25,251 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:56:25,253 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:25,253 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:25,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:25,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:25,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:25,253 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:25,253 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:56:25,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:25,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:25,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:25,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:25,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:25,254 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:56:25,254 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@71e95808[Count = 0] remaining members to acquire global barrier 2023-04-19 20:56:25,254 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:56:25,254 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:25,255 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:25,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:25,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:25,256 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:56:25,256 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:56:25,256 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:25,256 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:56:25,256 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:56:25,257 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:25,257 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:56:25,257 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:56:25,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:25,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:25,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:25,257 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:56:25,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:25,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:25,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:25,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:25,258 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:25,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:25,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57226 deadline: 1681937845257, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:25,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:25,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:25,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:56:25,259 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:56:25,259 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:56:25,259 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:56:25,260 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:56:25,260 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:56:25,260 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:56:25,261 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:25,261 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:25,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:25,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:25,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:25,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:25,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:25,261 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:56:25,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:25,262 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:25,262 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:56:25,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:56:25,263 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:25,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 306 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57236 deadline: 1681937845260, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:25,263 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:25,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57246 deadline: 1681937845260, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:25,265 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:25,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57204 deadline: 1681937845262, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:25,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:25,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:25,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:25,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:25,276 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:25,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:25,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:25,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:25,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:25,282 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:25,282 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:25,282 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:25,282 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:25,282 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:25,282 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:25,282 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:56:25,282 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:56:25,282 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:56:25,282 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:56:25,282 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:56:25,282 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:25,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:56:25,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:56:25,282 DEBUG [Thread-4043] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:56:25,282 DEBUG [Thread-4043] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:56:25,282 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:56:25,283 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:56:25,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:56:25,342 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:25,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57210 deadline: 1681937845337, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:25,343 DEBUG [Thread-4041] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4136 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267., hostname=jenkins-hbase4.apache.org,35247,1681937616971, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-19 20:56:25,351 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041928dc060a636540868b0f5f486d7cb5f9 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041928dc060a636540868b0f5f486d7cb5f9 2023-04-19 20:56:25,358 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=666, memsize=51.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/A/d37a0c52167b4628947ff053c9ea60cc 2023-04-19 20:56:25,366 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=666 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/B/bc60fa64bc764e56bff1892182c035fd 2023-04-19 20:56:25,377 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=666 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/C/37ef51f107644be5b36e45327b231d10 2023-04-19 20:56:25,382 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/A/d37a0c52167b4628947ff053c9ea60cc as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/d37a0c52167b4628947ff053c9ea60cc 2023-04-19 20:56:25,383 DEBUG [Thread-4043] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:56:25,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:56:25,386 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:56:25,386 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/d37a0c52167b4628947ff053c9ea60cc, entries=150, sequenceid=666, filesize=25.4 K 2023-04-19 20:56:25,387 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/B/bc60fa64bc764e56bff1892182c035fd as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/bc60fa64bc764e56bff1892182c035fd 2023-04-19 20:56:25,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:56:25,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:56:25,387 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:56:25,387 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:56:25,387 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:56:25,387 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:56:25,388 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:25,388 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:25,392 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:56:25,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:56:25,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:56:25,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:25,392 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:25,392 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:25,393 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:25,393 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:56:25,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:56:25,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:25,393 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:56:25,393 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:56:25,393 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:56:25,393 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:56:25,393 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:56:25,394 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:56:25,394 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:56:25,394 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:25,394 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. started... 2023-04-19 20:56:25,394 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:56:25,394 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. as already flushing 2023-04-19 20:56:25,394 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:25,394 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:56:25,394 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:56:25,394 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:25,394 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:56:25,394 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:56:25,395 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:25,396 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:25,396 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:25,397 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:56:25,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:25,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:25,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:25,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:25,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:25,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:25,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:25,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:25,398 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:56:25,398 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6649a240[Count = 0] remaining members to acquire global barrier 2023-04-19 20:56:25,398 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:56:25,399 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:25,401 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:25,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:25,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:25,401 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:56:25,401 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:56:25,401 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:56:25,402 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:25,402 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:56:25,403 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:56:25,403 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:56:25,403 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:56:25,403 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:25,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:25,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:25,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:25,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:25,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:25,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:25,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:25,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:25,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:25,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:25,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:56:25,406 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:56:25,406 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:56:25,406 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:56:25,406 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:56:25,406 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:56:25,406 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:56:25,408 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:25,408 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:56:25,408 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:25,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:25,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:56:25,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:56:25,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:25,409 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:25,409 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/bc60fa64bc764e56bff1892182c035fd, entries=150, sequenceid=666, filesize=11.9 K 2023-04-19 20:56:25,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:25,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:25,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:25,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:25,410 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:25,411 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/C/37ef51f107644be5b36e45327b231d10 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/37ef51f107644be5b36e45327b231d10 2023-04-19 20:56:25,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:25,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:25,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:25,412 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:25,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:25,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:25,424 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/37ef51f107644be5b36e45327b231d10, entries=150, sequenceid=666, filesize=11.9 K 2023-04-19 20:56:25,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:25,425 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~323.55 KB/331312, currentSize=67.09 KB/68700 for 6873ba93789e63dd9f14391208147267 in 487ms, sequenceid=666, compaction requested=true 2023-04-19 20:56:25,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:25,425 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 6873ba93789e63dd9f14391208147267: 2023-04-19 20:56:25,425 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 20:56:25,426 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 20:56:25,426 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-19 20:56:25,426 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-19 20:56:25,426 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-19 20:56:25,427 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:56:25,427 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:56:25,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:56:25,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:56:25,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:56:25,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:56:25,428 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:25,428 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:25,428 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:25,429 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:25,429 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:25,429 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:56:25,429 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:56:25,429 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:56:25,429 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:56:25,429 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:25,429 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:25,429 DEBUG [Thread-4043] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:56:25,429 DEBUG [Thread-4043] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:56:25,429 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 69286 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-19 20:56:25,429 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1912): 6873ba93789e63dd9f14391208147267/B is initiating minor compaction (all files) 2023-04-19 20:56:25,430 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6873ba93789e63dd9f14391208147267/B in TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:25,430 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/f48c5241a55842d6812e8404b9e4c5b2, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/12386e413b51429f85b6b262ee1b5e82, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/9efea23d519e445a9b3a894bdb478948, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/786395e45605458fb4c6447e2220cdd2, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/630dfd12107742b2a23a7044ee8e0d6e, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/bc60fa64bc764e56bff1892182c035fd] into tmpdir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp, totalSize=67.7 K 2023-04-19 20:56:25,430 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 150282 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-19 20:56:25,430 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting f48c5241a55842d6812e8404b9e4c5b2, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=494, earliestPutTs=1681937781208 2023-04-19 20:56:25,430 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1912): 6873ba93789e63dd9f14391208147267/A is initiating minor compaction (all files) 2023-04-19 20:56:25,430 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 6873ba93789e63dd9f14391208147267/A in TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:25,431 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/0c87feb59d8e4991ae7f1290e16b1c96, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/d04fedd90da34bb2a8a3d5a76415b76f, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/60d5476aad5f4856a5ec1f39d2188a1e, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/ed7f3b4bb0d84c5cabb2c08befee59fc, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/aee3852a71d64a3f9b9771d3f537012c, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/d37a0c52167b4628947ff053c9ea60cc] into tmpdir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp, totalSize=146.8 K 2023-04-19 20:56:25,431 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 12386e413b51429f85b6b262ee1b5e82, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=515, earliestPutTs=1681937781537 2023-04-19 20:56:25,431 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 0c87feb59d8e4991ae7f1290e16b1c96, keycount=150, bloomtype=ROW, size=26.5 K, encoding=NONE, compression=NONE, seqNum=494, earliestPutTs=1681937781208 2023-04-19 20:56:25,431 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 9efea23d519e445a9b3a894bdb478948, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=529, earliestPutTs=1681937782163 2023-04-19 20:56:25,432 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting d04fedd90da34bb2a8a3d5a76415b76f, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=515, earliestPutTs=1681937781537 2023-04-19 20:56:25,432 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 786395e45605458fb4c6447e2220cdd2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=548, earliestPutTs=1681937782752 2023-04-19 20:56:25,432 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 60d5476aad5f4856a5ec1f39d2188a1e, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=529, earliestPutTs=1681937782163 2023-04-19 20:56:25,432 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 630dfd12107742b2a23a7044ee8e0d6e, keycount=50, bloomtype=ROW, size=7.1 K, encoding=NONE, compression=NONE, seqNum=558, earliestPutTs=1681937783973 2023-04-19 20:56:25,432 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting ed7f3b4bb0d84c5cabb2c08befee59fc, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=548, earliestPutTs=1681937782752 2023-04-19 20:56:25,433 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting bc60fa64bc764e56bff1892182c035fd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=666, earliestPutTs=1681937784321 2023-04-19 20:56:25,434 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting aee3852a71d64a3f9b9771d3f537012c, keycount=100, bloomtype=ROW, size=18.6 K, encoding=NONE, compression=NONE, seqNum=558, earliestPutTs=1681937783960 2023-04-19 20:56:25,434 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting d37a0c52167b4628947ff053c9ea60cc, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=666, earliestPutTs=1681937784321 2023-04-19 20:56:25,446 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] throttle.PressureAwareThroughputController(145): 6873ba93789e63dd9f14391208147267#B#compaction#653 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 20:56:25,465 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] throttle.PressureAwareThroughputController(145): 6873ba93789e63dd9f14391208147267#A#compaction#654 average throughput is 1.16 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 20:56:25,529 DEBUG [Thread-4043] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:56:25,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:56:25,531 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:56:25,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:56:25,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:56:25,533 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:56:25,533 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:56:25,533 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:56:25,533 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:56:25,533 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:25,533 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:25,538 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:56:25,538 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:25,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:56:25,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:56:25,539 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:25,539 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:56:25,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:25,539 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:25,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:56:25,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:25,540 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:56:25,540 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:56:25,540 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:56:25,540 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:56:25,540 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:56:25,540 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:56:25,540 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:56:25,540 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:56:25,540 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:25,541 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. started... 2023-04-19 20:56:25,541 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6873ba93789e63dd9f14391208147267 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-19 20:56:25,541 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6873ba93789e63dd9f14391208147267, store=A 2023-04-19 20:56:25,541 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:56:25,541 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6873ba93789e63dd9f14391208147267, store=B 2023-04-19 20:56:25,541 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:56:25,541 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6873ba93789e63dd9f14391208147267, store=C 2023-04-19 20:56:25,541 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:56:25,559 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304199cb9fc35d1f640d1a908890300190c85 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304199cb9fc35d1f640d1a908890300190c85 2023-04-19 20:56:25,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(9158): Flush requested on 6873ba93789e63dd9f14391208147267 2023-04-19 20:56:25,562 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. as already flushing 2023-04-19 20:56:25,564 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=679, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/A/6ca70a52f4a7454b9a4ce9efbc5a0142 2023-04-19 20:56:25,574 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=679 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/B/85e2b51df81e4fecadd6ab1b19338e36 2023-04-19 20:56:25,590 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:25,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57204 deadline: 1681937845584, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:25,595 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:25,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 235 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57226 deadline: 1681937845589, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:25,596 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:25,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 312 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57236 deadline: 1681937845590, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:25,596 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:25,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57246 deadline: 1681937845591, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:25,692 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:25,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57204 deadline: 1681937845691, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:25,698 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:25,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 237 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57226 deadline: 1681937845696, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:25,699 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:25,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 314 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57236 deadline: 1681937845697, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:25,699 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:25,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 232 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57246 deadline: 1681937845697, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:25,865 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/B/21ca76f476b94c1ab02eb19695831e22 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/21ca76f476b94c1ab02eb19695831e22 2023-04-19 20:56:25,869 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 6873ba93789e63dd9f14391208147267/B of 6873ba93789e63dd9f14391208147267 into 21ca76f476b94c1ab02eb19695831e22(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 20:56:25,869 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6873ba93789e63dd9f14391208147267: 2023-04-19 20:56:25,869 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267., storeName=6873ba93789e63dd9f14391208147267/B, priority=10, startTime=1681937785426; duration=0sec 2023-04-19 20:56:25,869 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 20:56:25,869 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-19 20:56:25,871 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 69252 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-19 20:56:25,871 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1912): 6873ba93789e63dd9f14391208147267/C is initiating minor compaction (all files) 2023-04-19 20:56:25,871 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6873ba93789e63dd9f14391208147267/C in TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:25,871 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/3b0a0c2531804a32bb02046673d6ba7e, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/aa108859b8444a58bc53bfe600264938, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/f1a7cebb8d78421081faf72f3fef352e, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/2b4460932e38440099225f3c7589f7c1, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/ae53b4179b724aa888e021e60fb3c784, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/37ef51f107644be5b36e45327b231d10] into tmpdir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp, totalSize=67.6 K 2023-04-19 20:56:25,871 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 3b0a0c2531804a32bb02046673d6ba7e, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=494, earliestPutTs=1681937781208 2023-04-19 20:56:25,873 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting aa108859b8444a58bc53bfe600264938, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=515, earliestPutTs=1681937781537 2023-04-19 20:56:25,873 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting f1a7cebb8d78421081faf72f3fef352e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=529, earliestPutTs=1681937782163 2023-04-19 20:56:25,873 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 2b4460932e38440099225f3c7589f7c1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=548, earliestPutTs=1681937782752 2023-04-19 20:56:25,873 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting ae53b4179b724aa888e021e60fb3c784, keycount=50, bloomtype=ROW, size=7.1 K, encoding=NONE, compression=NONE, seqNum=558, earliestPutTs=1681937783973 2023-04-19 20:56:25,874 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 37ef51f107644be5b36e45327b231d10, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=666, earliestPutTs=1681937784321 2023-04-19 20:56:25,882 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] throttle.PressureAwareThroughputController(145): 6873ba93789e63dd9f14391208147267#C#compaction#658 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 20:56:25,896 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/C/b1081035997f4838af62a1c177db5427 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/b1081035997f4838af62a1c177db5427 2023-04-19 20:56:25,897 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:25,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57204 deadline: 1681937845894, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:25,899 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/A/a701b2164253433fbe6d810232758682 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/a701b2164253433fbe6d810232758682 2023-04-19 20:56:25,901 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 6873ba93789e63dd9f14391208147267/C of 6873ba93789e63dd9f14391208147267 into b1081035997f4838af62a1c177db5427(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 20:56:25,901 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6873ba93789e63dd9f14391208147267: 2023-04-19 20:56:25,901 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267., storeName=6873ba93789e63dd9f14391208147267/C, priority=10, startTime=1681937785426; duration=0sec 2023-04-19 20:56:25,901 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 20:56:25,903 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:25,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 316 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57236 deadline: 1681937845900, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:25,903 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:25,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57226 deadline: 1681937845900, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:25,903 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:25,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57246 deadline: 1681937845901, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:25,903 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 6873ba93789e63dd9f14391208147267/A of 6873ba93789e63dd9f14391208147267 into a701b2164253433fbe6d810232758682(size=26.7 K), total size for store is 26.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 20:56:25,904 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6873ba93789e63dd9f14391208147267: 2023-04-19 20:56:25,904 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267., storeName=6873ba93789e63dd9f14391208147267/A, priority=10, startTime=1681937785425; duration=0sec 2023-04-19 20:56:25,904 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 20:56:25,984 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=679 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/C/9f499bfde60447bb80a012903f69fe0d 2023-04-19 20:56:25,988 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/A/6ca70a52f4a7454b9a4ce9efbc5a0142 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/6ca70a52f4a7454b9a4ce9efbc5a0142 2023-04-19 20:56:25,991 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/6ca70a52f4a7454b9a4ce9efbc5a0142, entries=150, sequenceid=679, filesize=25.4 K 2023-04-19 20:56:25,992 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/B/85e2b51df81e4fecadd6ab1b19338e36 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/85e2b51df81e4fecadd6ab1b19338e36 2023-04-19 20:56:25,996 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/85e2b51df81e4fecadd6ab1b19338e36, entries=150, sequenceid=679, filesize=11.9 K 2023-04-19 20:56:25,996 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/C/9f499bfde60447bb80a012903f69fe0d as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/9f499bfde60447bb80a012903f69fe0d 2023-04-19 20:56:25,999 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/9f499bfde60447bb80a012903f69fe0d, entries=150, sequenceid=679, filesize=11.9 K 2023-04-19 20:56:25,999 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=114.05 KB/116790 for 6873ba93789e63dd9f14391208147267 in 458ms, sequenceid=679, compaction requested=false 2023-04-19 20:56:26,000 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6873ba93789e63dd9f14391208147267: 2023-04-19 20:56:26,000 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:26,000 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:56:26,000 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:56:26,000 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:26,000 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:56:26,000 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:56:26,002 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:26,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:26,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:26,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:26,002 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:26,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:26,003 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:26,003 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:56:26,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:26,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:26,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:26,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:26,004 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:56:26,004 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@21064cbb[Count = 0] remaining members to acquire global barrier 2023-04-19 20:56:26,004 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:56:26,005 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:26,006 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:26,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:26,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:26,006 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:56:26,006 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:56:26,006 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:56:26,006 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:26,006 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:56:26,008 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:56:26,008 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:26,008 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:56:26,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:26,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:26,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:26,008 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:56:26,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:26,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:26,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:26,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:26,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:26,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:26,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:26,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:56:26,011 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:56:26,011 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:56:26,011 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:56:26,011 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:56:26,011 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:56:26,011 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:56:26,013 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:26,013 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:56:26,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:26,013 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:26,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:26,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:26,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:26,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:26,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:56:26,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:56:26,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:26,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:26,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:26,014 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:26,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:26,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:26,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:26,015 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:26,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:26,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:26,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:26,018 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:56:26,018 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:56:26,018 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:26,018 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:56:26,018 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:26,018 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:56:26,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:56:26,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:56:26,018 DEBUG [Thread-4043] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:56:26,018 DEBUG [Thread-4043] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:56:26,018 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:56:26,018 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:26,018 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:26,018 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:26,018 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:26,018 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:26,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:56:26,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:56:26,118 DEBUG [Thread-4043] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:56:26,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:56:26,121 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 20:56:26,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 20:56:26,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 20:56:26,122 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 20:56:26,122 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:56:26,122 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 20:56:26,122 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 20:56:26,122 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:26,123 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:26,124 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:56:26,124 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:26,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:56:26,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:56:26,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:26,124 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:26,124 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 20:56:26,124 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:26,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 20:56:26,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:26,125 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 20:56:26,125 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 20:56:26,125 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 20:56:26,125 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 20:56:26,125 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 20:56:26,125 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 20:56:26,125 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 20:56:26,125 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 20:56:26,125 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:26,126 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. started... 2023-04-19 20:56:26,126 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6873ba93789e63dd9f14391208147267 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-19 20:56:26,126 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6873ba93789e63dd9f14391208147267, store=A 2023-04-19 20:56:26,126 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:56:26,126 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6873ba93789e63dd9f14391208147267, store=B 2023-04-19 20:56:26,126 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:56:26,126 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6873ba93789e63dd9f14391208147267, store=C 2023-04-19 20:56:26,126 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:56:26,145 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041951c6ec6a80374a51a0d107266ade06a6 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041951c6ec6a80374a51a0d107266ade06a6 2023-04-19 20:56:26,150 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=702, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/A/3ad416fa93f84a8d88acdf508a8c63f6 2023-04-19 20:56:26,159 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=702 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/B/08945219adc64bb4923f4ff735a3e709 2023-04-19 20:56:26,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(9158): Flush requested on 6873ba93789e63dd9f14391208147267 2023-04-19 20:56:26,203 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. as already flushing 2023-04-19 20:56:26,213 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:26,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 242 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57226 deadline: 1681937846210, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:26,216 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:26,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57246 deadline: 1681937846213, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:26,216 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:26,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 320 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57236 deadline: 1681937846213, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:26,216 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:26,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57204 deadline: 1681937846213, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:26,316 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:26,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 244 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57226 deadline: 1681937846314, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:26,318 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:26,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 322 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57236 deadline: 1681937846317, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:26,318 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:26,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57246 deadline: 1681937846317, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:26,320 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:26,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35247] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57204 deadline: 1681937846317, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:26,442 DEBUG [Thread-4044] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3c1a68cd to 127.0.0.1:58736 2023-04-19 20:56:26,442 DEBUG [Thread-4052] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5fdfef3b to 127.0.0.1:58736 2023-04-19 20:56:26,442 DEBUG [Thread-4044] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-19 20:56:26,443 DEBUG [Thread-4052] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-19 20:56:26,445 DEBUG [Thread-4046] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5a709b21 to 127.0.0.1:58736 2023-04-19 20:56:26,445 DEBUG [Thread-4046] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-19 20:56:26,445 DEBUG [Thread-4048] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x156e4ba6 to 127.0.0.1:58736 2023-04-19 20:56:26,445 DEBUG [Thread-4048] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-19 20:56:26,445 DEBUG [Thread-4050] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x785e366b to 127.0.0.1:58736 2023-04-19 20:56:26,445 DEBUG [Thread-4050] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-19 20:56:26,518 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:26,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 246 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57226 deadline: 1681937846518, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:26,519 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:26,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35247] ipc.CallRunner(144): callId: 324 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57236 deadline: 1681937846519, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:26,519 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:26,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 242 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57246 deadline: 1681937846519, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:26,521 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 20:56:26,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57204 deadline: 1681937846521, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:26,568 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=702 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/C/255f7a2e44bb4d44bef6296297a953fe 2023-04-19 20:56:26,571 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/A/3ad416fa93f84a8d88acdf508a8c63f6 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/3ad416fa93f84a8d88acdf508a8c63f6 2023-04-19 20:56:26,574 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/3ad416fa93f84a8d88acdf508a8c63f6, entries=150, sequenceid=702, filesize=25.4 K 2023-04-19 20:56:26,574 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/B/08945219adc64bb4923f4ff735a3e709 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/08945219adc64bb4923f4ff735a3e709 2023-04-19 20:56:26,577 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/08945219adc64bb4923f4ff735a3e709, entries=150, sequenceid=702, filesize=11.9 K 2023-04-19 20:56:26,578 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/C/255f7a2e44bb4d44bef6296297a953fe as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/255f7a2e44bb4d44bef6296297a953fe 2023-04-19 20:56:26,580 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/255f7a2e44bb4d44bef6296297a953fe, entries=150, sequenceid=702, filesize=11.9 K 2023-04-19 20:56:26,581 INFO [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 6873ba93789e63dd9f14391208147267 in 455ms, sequenceid=702, compaction requested=true 2023-04-19 20:56:26,581 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6873ba93789e63dd9f14391208147267: 2023-04-19 20:56:26,581 DEBUG [rs(jenkins-hbase4.apache.org,35247,1681937616971)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:26,581 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 20:56:26,581 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 20:56:26,581 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:26,581 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 20:56:26,581 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 20:56:26,583 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:26,583 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:26,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:26,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:26,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:26,583 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:26,583 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 20:56:26,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:26,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:26,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:26,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:26,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:26,584 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35247,1681937616971' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 20:56:26,584 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3b967434[Count = 0] remaining members to acquire global barrier 2023-04-19 20:56:26,584 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 20:56:26,584 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:26,585 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:26,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:26,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:26,585 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 20:56:26,585 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 20:56:26,585 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35247,1681937616971' in zk 2023-04-19 20:56:26,585 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:26,585 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 20:56:26,587 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:26,587 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 20:56:26,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:26,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:26,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:26,587 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:56:26,587 DEBUG [member: 'jenkins-hbase4.apache.org,35247,1681937616971' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 20:56:26,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:26,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:26,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:26,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:26,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:26,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:26,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:26,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35247,1681937616971': 2023-04-19 20:56:26,589 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35247,1681937616971' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 20:56:26,589 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 20:56:26,589 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 20:56:26,589 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 20:56:26,589 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 20:56:26,589 INFO [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 20:56:26,590 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:26,590 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:26,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:26,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:26,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 20:56:26,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 20:56:26,590 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:56:26,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:26,591 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:26,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:56:26,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:56:26,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 20:56:26,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:26,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:26,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 20:56:26,592 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:26,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:26,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:26,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 20:56:26,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 20:56:26,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:26,593 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:26,593 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 20:56:26,593 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:26,594 DEBUG [(jenkins-hbase4.apache.org,37181,1681937615733)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 20:56:26,594 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 20:56:26,594 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 20:56:26,594 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 20:56:26,594 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:26,594 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:26,594 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 20:56:26,594 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 20:56:26,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 20:56:26,594 DEBUG [Thread-4043] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 20:56:26,594 DEBUG [Thread-4043] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 20:56:26,593 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 20:56:26,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 20:56:26,594 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 20:56:26,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 20:56:26,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 20:56:26,694 DEBUG [Thread-4043] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 20:56:26,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 20:56:26,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35247] regionserver.HRegion(9158): Flush requested on 6873ba93789e63dd9f14391208147267 2023-04-19 20:56:26,822 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 6873ba93789e63dd9f14391208147267 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-19 20:56:26,822 DEBUG [Thread-4033] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6257294f to 127.0.0.1:58736 2023-04-19 20:56:26,822 DEBUG [Thread-4033] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-19 20:56:26,822 DEBUG [Thread-4037] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4a320817 to 127.0.0.1:58736 2023-04-19 20:56:26,822 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6873ba93789e63dd9f14391208147267, store=A 2023-04-19 20:56:26,822 DEBUG [Thread-4037] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-19 20:56:26,822 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:56:26,822 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6873ba93789e63dd9f14391208147267, store=B 2023-04-19 20:56:26,822 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:56:26,822 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6873ba93789e63dd9f14391208147267, store=C 2023-04-19 20:56:26,822 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:56:26,838 DEBUG [Thread-4039] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x19c1e3ee to 127.0.0.1:58736 2023-04-19 20:56:26,838 DEBUG [Thread-4035] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x52104714 to 127.0.0.1:58736 2023-04-19 20:56:26,838 DEBUG [Thread-4039] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-19 20:56:26,839 DEBUG [Thread-4035] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-19 20:56:27,232 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304191520ea61b5964207b939a9b006f9f0c3 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304191520ea61b5964207b939a9b006f9f0c3 2023-04-19 20:56:27,236 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=716, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/A/db1a88079b4c495a8a7aaf0dc0430983 2023-04-19 20:56:27,644 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=716 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/B/f2a3025aff7b4a5d9c309a18448868af 2023-04-19 20:56:28,052 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=716 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/C/658f4875e9e64d7c87638411cc43601c 2023-04-19 20:56:28,055 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/A/db1a88079b4c495a8a7aaf0dc0430983 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/db1a88079b4c495a8a7aaf0dc0430983 2023-04-19 20:56:28,058 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/db1a88079b4c495a8a7aaf0dc0430983, entries=150, sequenceid=716, filesize=25.4 K 2023-04-19 20:56:28,058 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/B/f2a3025aff7b4a5d9c309a18448868af as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/f2a3025aff7b4a5d9c309a18448868af 2023-04-19 20:56:28,061 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/f2a3025aff7b4a5d9c309a18448868af, entries=150, sequenceid=716, filesize=11.9 K 2023-04-19 20:56:28,061 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/C/658f4875e9e64d7c87638411cc43601c as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/658f4875e9e64d7c87638411cc43601c 2023-04-19 20:56:28,064 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/658f4875e9e64d7c87638411cc43601c, entries=150, sequenceid=716, filesize=11.9 K 2023-04-19 20:56:28,064 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=13.42 KB/13740 for 6873ba93789e63dd9f14391208147267 in 1242ms, sequenceid=716, compaction requested=true 2023-04-19 20:56:28,065 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 6873ba93789e63dd9f14391208147267: 2023-04-19 20:56:28,065 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 20:56:28,065 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 20:56:28,065 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-19 20:56:28,065 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-19 20:56:28,065 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 20:56:28,066 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 105444 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-19 20:56:28,066 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 50004 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-19 20:56:28,066 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1912): 6873ba93789e63dd9f14391208147267/A is initiating minor compaction (all files) 2023-04-19 20:56:28,066 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1912): 6873ba93789e63dd9f14391208147267/B is initiating minor compaction (all files) 2023-04-19 20:56:28,066 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 6873ba93789e63dd9f14391208147267/A in TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:28,066 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6873ba93789e63dd9f14391208147267/B in TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:28,066 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/a701b2164253433fbe6d810232758682, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/6ca70a52f4a7454b9a4ce9efbc5a0142, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/3ad416fa93f84a8d88acdf508a8c63f6, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/db1a88079b4c495a8a7aaf0dc0430983] into tmpdir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp, totalSize=103.0 K 2023-04-19 20:56:28,066 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/21ca76f476b94c1ab02eb19695831e22, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/85e2b51df81e4fecadd6ab1b19338e36, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/08945219adc64bb4923f4ff735a3e709, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/f2a3025aff7b4a5d9c309a18448868af] into tmpdir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp, totalSize=48.8 K 2023-04-19 20:56:28,066 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting a701b2164253433fbe6d810232758682, keycount=150, bloomtype=ROW, size=26.7 K, encoding=NONE, compression=NONE, seqNum=666, earliestPutTs=1681937784321 2023-04-19 20:56:28,066 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 21ca76f476b94c1ab02eb19695831e22, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=666, earliestPutTs=1681937784321 2023-04-19 20:56:28,067 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 6ca70a52f4a7454b9a4ce9efbc5a0142, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=679, earliestPutTs=1681937784940 2023-04-19 20:56:28,067 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 85e2b51df81e4fecadd6ab1b19338e36, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=679, earliestPutTs=1681937784940 2023-04-19 20:56:28,067 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting 3ad416fa93f84a8d88acdf508a8c63f6, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=702, earliestPutTs=1681937785583 2023-04-19 20:56:28,067 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 08945219adc64bb4923f4ff735a3e709, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=702, earliestPutTs=1681937785583 2023-04-19 20:56:28,067 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] compactions.Compactor(207): Compacting db1a88079b4c495a8a7aaf0dc0430983, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=716, earliestPutTs=1681937786211 2023-04-19 20:56:28,067 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting f2a3025aff7b4a5d9c309a18448868af, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=716, earliestPutTs=1681937786211 2023-04-19 20:56:28,080 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] throttle.PressureAwareThroughputController(145): 6873ba93789e63dd9f14391208147267#B#compaction#666 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-19 20:56:28,081 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] throttle.PressureAwareThroughputController(145): 6873ba93789e63dd9f14391208147267#A#compaction#665 average throughput is 6.57 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 20:56:28,487 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/B/e0c6e1e4e1b24ee6b88863dc4f208b17 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/e0c6e1e4e1b24ee6b88863dc4f208b17 2023-04-19 20:56:28,491 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 6873ba93789e63dd9f14391208147267/B of 6873ba93789e63dd9f14391208147267 into e0c6e1e4e1b24ee6b88863dc4f208b17(size=13.3 K), total size for store is 13.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 20:56:28,491 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6873ba93789e63dd9f14391208147267: 2023-04-19 20:56:28,491 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267., storeName=6873ba93789e63dd9f14391208147267/B, priority=12, startTime=1681937788065; duration=0sec 2023-04-19 20:56:28,492 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 20:56:28,492 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-19 20:56:28,493 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49970 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-19 20:56:28,493 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1912): 6873ba93789e63dd9f14391208147267/C is initiating minor compaction (all files) 2023-04-19 20:56:28,493 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6873ba93789e63dd9f14391208147267/C in TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:28,493 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/b1081035997f4838af62a1c177db5427, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/9f499bfde60447bb80a012903f69fe0d, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/255f7a2e44bb4d44bef6296297a953fe, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/658f4875e9e64d7c87638411cc43601c] into tmpdir=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp, totalSize=48.8 K 2023-04-19 20:56:28,493 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting b1081035997f4838af62a1c177db5427, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=666, earliestPutTs=1681937784321 2023-04-19 20:56:28,493 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 9f499bfde60447bb80a012903f69fe0d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=679, earliestPutTs=1681937784940 2023-04-19 20:56:28,494 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 255f7a2e44bb4d44bef6296297a953fe, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=702, earliestPutTs=1681937785583 2023-04-19 20:56:28,494 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] compactions.Compactor(207): Compacting 658f4875e9e64d7c87638411cc43601c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=716, earliestPutTs=1681937786211 2023-04-19 20:56:28,499 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/A/eb4d9bb777fd4deaa3e44a1e77607fcb as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/eb4d9bb777fd4deaa3e44a1e77607fcb 2023-04-19 20:56:28,503 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 6873ba93789e63dd9f14391208147267/A of 6873ba93789e63dd9f14391208147267 into eb4d9bb777fd4deaa3e44a1e77607fcb(size=26.8 K), total size for store is 26.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 20:56:28,503 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6873ba93789e63dd9f14391208147267: 2023-04-19 20:56:28,503 INFO [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267., storeName=6873ba93789e63dd9f14391208147267/A, priority=12, startTime=1681937788065; duration=0sec 2023-04-19 20:56:28,503 DEBUG [RS:0;jenkins-hbase4:35247-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 20:56:28,503 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] throttle.PressureAwareThroughputController(145): 6873ba93789e63dd9f14391208147267#C#compaction#667 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 20:56:28,910 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/C/d57f9f95bf8647c0a4bef1f7d0b18692 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/d57f9f95bf8647c0a4bef1f7d0b18692 2023-04-19 20:56:28,913 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 6873ba93789e63dd9f14391208147267/C of 6873ba93789e63dd9f14391208147267 into d57f9f95bf8647c0a4bef1f7d0b18692(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 20:56:28,913 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6873ba93789e63dd9f14391208147267: 2023-04-19 20:56:28,913 INFO [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267., storeName=6873ba93789e63dd9f14391208147267/C, priority=12, startTime=1681937788065; duration=0sec 2023-04-19 20:56:28,913 DEBUG [RS:0;jenkins-hbase4:35247-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 20:56:29,359 DEBUG [Thread-4041] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x526f2221 to 127.0.0.1:58736 2023-04-19 20:56:29,359 DEBUG [Thread-4041] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-19 20:56:29,359 INFO [Listener at localhost/34893] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-19 20:56:29,359 INFO [Listener at localhost/34893] hbase.AcidGuaranteesTestTool(394): wrote 101 2023-04-19 20:56:29,359 INFO [Listener at localhost/34893] hbase.AcidGuaranteesTestTool(394): wrote 107 2023-04-19 20:56:29,359 INFO [Listener at localhost/34893] hbase.AcidGuaranteesTestTool(394): wrote 97 2023-04-19 20:56:29,359 INFO [Listener at localhost/34893] hbase.AcidGuaranteesTestTool(394): wrote 187 2023-04-19 20:56:29,359 INFO [Listener at localhost/34893] hbase.AcidGuaranteesTestTool(394): wrote 98 2023-04-19 20:56:29,360 INFO [Listener at localhost/34893] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-19 20:56:29,360 INFO [Listener at localhost/34893] hbase.AcidGuaranteesTestTool(398): read 4424 2023-04-19 20:56:29,360 INFO [Listener at localhost/34893] hbase.AcidGuaranteesTestTool(398): read 4336 2023-04-19 20:56:29,360 INFO [Listener at localhost/34893] hbase.AcidGuaranteesTestTool(398): read 4341 2023-04-19 20:56:29,360 INFO [Listener at localhost/34893] hbase.AcidGuaranteesTestTool(398): read 4437 2023-04-19 20:56:29,360 INFO [Listener at localhost/34893] hbase.AcidGuaranteesTestTool(398): read 4349 2023-04-19 20:56:29,360 INFO [Listener at localhost/34893] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-19 20:56:29,360 INFO [Listener at localhost/34893] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-19 20:56:29,360 DEBUG [Listener at localhost/34893] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0c0766f0 to 127.0.0.1:58736 2023-04-19 20:56:29,360 DEBUG [Listener at localhost/34893] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-19 20:56:29,361 INFO [Listener at localhost/34893] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-19 20:56:29,361 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-19 20:56:29,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure2.ProcedureExecutor(1029): Stored pid=62, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-19 20:56:29,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-19 20:56:29,364 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681937789364"}]},"ts":"1681937789364"} 2023-04-19 20:56:29,365 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-19 20:56:29,366 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-19 20:56:29,366 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=63, ppid=62, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=6873ba93789e63dd9f14391208147267, UNASSIGN}] 2023-04-19 20:56:29,368 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=63, ppid=62, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=6873ba93789e63dd9f14391208147267, UNASSIGN 2023-04-19 20:56:29,368 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=63 updating hbase:meta row=6873ba93789e63dd9f14391208147267, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:29,368 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681937789368"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681937789368"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681937789368"}]},"ts":"1681937789368"} 2023-04-19 20:56:29,369 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=64, ppid=63, state=RUNNABLE; CloseRegionProcedure 6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971}] 2023-04-19 20:56:29,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-19 20:56:29,520 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 6873ba93789e63dd9f14391208147267 2023-04-19 20:56:29,521 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 6873ba93789e63dd9f14391208147267, disabling compactions & flushes 2023-04-19 20:56:29,521 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:29,521 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:29,521 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. after waiting 0 ms 2023-04-19 20:56:29,521 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:29,521 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 6873ba93789e63dd9f14391208147267 3/3 column families, dataSize=20.13 KB heapSize=60.52 KB 2023-04-19 20:56:29,521 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6873ba93789e63dd9f14391208147267, store=A 2023-04-19 20:56:29,521 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:56:29,521 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6873ba93789e63dd9f14391208147267, store=B 2023-04-19 20:56:29,521 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:56:29,521 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6873ba93789e63dd9f14391208147267, store=C 2023-04-19 20:56:29,521 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 20:56:29,529 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230419ad27124eaf3849e18338f58fda8fd303 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419ad27124eaf3849e18338f58fda8fd303 2023-04-19 20:56:29,533 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=725, memsize=6.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/A/f6368fc7393d40f883e4513f67d53cb9 2023-04-19 20:56:29,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-19 20:56:29,941 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=725 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/B/fc97ee9a48494d448a63b5d093856aa9 2023-04-19 20:56:29,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-19 20:56:30,349 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=725 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/C/0f0730f31db44bf6bbcadb132cc01a31 2023-04-19 20:56:30,352 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/A/f6368fc7393d40f883e4513f67d53cb9 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/f6368fc7393d40f883e4513f67d53cb9 2023-04-19 20:56:30,355 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/f6368fc7393d40f883e4513f67d53cb9, entries=100, sequenceid=725, filesize=18.6 K 2023-04-19 20:56:30,355 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/B/fc97ee9a48494d448a63b5d093856aa9 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/fc97ee9a48494d448a63b5d093856aa9 2023-04-19 20:56:30,358 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/fc97ee9a48494d448a63b5d093856aa9, entries=100, sequenceid=725, filesize=9.5 K 2023-04-19 20:56:30,359 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/.tmp/C/0f0730f31db44bf6bbcadb132cc01a31 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/0f0730f31db44bf6bbcadb132cc01a31 2023-04-19 20:56:30,361 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/0f0730f31db44bf6bbcadb132cc01a31, entries=100, sequenceid=725, filesize=9.5 K 2023-04-19 20:56:30,362 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.13 KB/20610, heapSize ~60.47 KB/61920, currentSize=0 B/0 for 6873ba93789e63dd9f14391208147267 in 841ms, sequenceid=725, compaction requested=false 2023-04-19 20:56:30,384 DEBUG [StoreCloser-TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/ab63796c353f4994a363f7bc53116b16, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/862c0865cb78432f8b2f2bf0bd08d25f, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/eec6a4efa75342c8aaa239f120fc5773, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/f36f8c2b86634c97b8ffb1851c04b27a, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/8f95ea25964a4a5cb50534c22f1f2dfb, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/f8f1c07250574ed481e0dca56a8de79c, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/3fa83711bf924f2f80a381c4fa673d33, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/38a4c874c52a4ed99810d21c4dee83a5, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/c8a7518ef45b49388d22210fd2f92996, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/524ff811bb4f44098edc0857a6061a60, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/ddc21f4dd5f64c789829525f787ae63b, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/350e37412298417db2ae5e7654af6c20, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/ccce7b41508b4527bdf147853f2446dc, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/adde6acef56446f18b37a68250af7b25, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/c0c129003c1e49ef96f8542c0e4b0b16, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/4c632f2df9dc4efbb2aecf0c495aa7f5, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/d5f423d3d0d54908a0da617f46dee134, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/8e1a60905ca2404d90b6b0533e03aaa0, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/113fc71247004215b1dca5f5e79eec30, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/837d84599682422d90be5bddf2164f89, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/301e9045dce443a7bfaa2b07a2816403, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/607f4960866b4b768437363b25dd3849, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/e32da98b4c9f4f4194b62ee10e46e168, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/deb0d77a660046108ba1ce6d3d9e3396, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/b5728a88b2374867a4f43c817e80f21b, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/c2c5846bd82740d0ac3d4ad8028faf3f, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/579c2d90b9a14427aa382495e3b7e88c, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/11b3f4bb7946432ca51e9ceaa3667218, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/90e84052bff54e508728bfe237e575ed, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/37ab81a43c16427e877a21033f390d02, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/151c342063d7435c960dcd3d8eb1c2a5, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/0c87feb59d8e4991ae7f1290e16b1c96, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/d04fedd90da34bb2a8a3d5a76415b76f, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/60d5476aad5f4856a5ec1f39d2188a1e, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/ed7f3b4bb0d84c5cabb2c08befee59fc, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/aee3852a71d64a3f9b9771d3f537012c, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/a701b2164253433fbe6d810232758682, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/d37a0c52167b4628947ff053c9ea60cc, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/6ca70a52f4a7454b9a4ce9efbc5a0142, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/3ad416fa93f84a8d88acdf508a8c63f6, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/db1a88079b4c495a8a7aaf0dc0430983] to archive 2023-04-19 20:56:30,385 DEBUG [StoreCloser-TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-19 20:56:30,386 DEBUG [StoreCloser-TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/ab63796c353f4994a363f7bc53116b16 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/ab63796c353f4994a363f7bc53116b16 2023-04-19 20:56:30,387 DEBUG [StoreCloser-TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/862c0865cb78432f8b2f2bf0bd08d25f to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/862c0865cb78432f8b2f2bf0bd08d25f 2023-04-19 20:56:30,388 DEBUG [StoreCloser-TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/eec6a4efa75342c8aaa239f120fc5773 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/eec6a4efa75342c8aaa239f120fc5773 2023-04-19 20:56:30,389 DEBUG [StoreCloser-TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/f36f8c2b86634c97b8ffb1851c04b27a to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/f36f8c2b86634c97b8ffb1851c04b27a 2023-04-19 20:56:30,390 DEBUG [StoreCloser-TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/8f95ea25964a4a5cb50534c22f1f2dfb to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/8f95ea25964a4a5cb50534c22f1f2dfb 2023-04-19 20:56:30,391 DEBUG [StoreCloser-TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/f8f1c07250574ed481e0dca56a8de79c to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/f8f1c07250574ed481e0dca56a8de79c 2023-04-19 20:56:30,392 DEBUG [StoreCloser-TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/3fa83711bf924f2f80a381c4fa673d33 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/3fa83711bf924f2f80a381c4fa673d33 2023-04-19 20:56:30,392 DEBUG [StoreCloser-TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/38a4c874c52a4ed99810d21c4dee83a5 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/38a4c874c52a4ed99810d21c4dee83a5 2023-04-19 20:56:30,393 DEBUG [StoreCloser-TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/c8a7518ef45b49388d22210fd2f92996 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/c8a7518ef45b49388d22210fd2f92996 2023-04-19 20:56:30,394 DEBUG [StoreCloser-TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/524ff811bb4f44098edc0857a6061a60 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/524ff811bb4f44098edc0857a6061a60 2023-04-19 20:56:30,395 DEBUG [StoreCloser-TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/ddc21f4dd5f64c789829525f787ae63b to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/ddc21f4dd5f64c789829525f787ae63b 2023-04-19 20:56:30,395 DEBUG [StoreCloser-TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/350e37412298417db2ae5e7654af6c20 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/350e37412298417db2ae5e7654af6c20 2023-04-19 20:56:30,396 DEBUG [StoreCloser-TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/ccce7b41508b4527bdf147853f2446dc to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/ccce7b41508b4527bdf147853f2446dc 2023-04-19 20:56:30,397 DEBUG [StoreCloser-TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/adde6acef56446f18b37a68250af7b25 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/adde6acef56446f18b37a68250af7b25 2023-04-19 20:56:30,397 DEBUG [StoreCloser-TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/c0c129003c1e49ef96f8542c0e4b0b16 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/c0c129003c1e49ef96f8542c0e4b0b16 2023-04-19 20:56:30,398 DEBUG [StoreCloser-TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/4c632f2df9dc4efbb2aecf0c495aa7f5 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/4c632f2df9dc4efbb2aecf0c495aa7f5 2023-04-19 20:56:30,399 DEBUG [StoreCloser-TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/d5f423d3d0d54908a0da617f46dee134 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/d5f423d3d0d54908a0da617f46dee134 2023-04-19 20:56:30,400 DEBUG [StoreCloser-TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/8e1a60905ca2404d90b6b0533e03aaa0 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/8e1a60905ca2404d90b6b0533e03aaa0 2023-04-19 20:56:30,400 DEBUG [StoreCloser-TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/113fc71247004215b1dca5f5e79eec30 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/113fc71247004215b1dca5f5e79eec30 2023-04-19 20:56:30,401 DEBUG [StoreCloser-TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/837d84599682422d90be5bddf2164f89 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/837d84599682422d90be5bddf2164f89 2023-04-19 20:56:30,402 DEBUG [StoreCloser-TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/301e9045dce443a7bfaa2b07a2816403 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/301e9045dce443a7bfaa2b07a2816403 2023-04-19 20:56:30,403 DEBUG [StoreCloser-TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/607f4960866b4b768437363b25dd3849 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/607f4960866b4b768437363b25dd3849 2023-04-19 20:56:30,404 DEBUG [StoreCloser-TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/e32da98b4c9f4f4194b62ee10e46e168 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/e32da98b4c9f4f4194b62ee10e46e168 2023-04-19 20:56:30,404 DEBUG [StoreCloser-TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/deb0d77a660046108ba1ce6d3d9e3396 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/deb0d77a660046108ba1ce6d3d9e3396 2023-04-19 20:56:30,405 DEBUG [StoreCloser-TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/b5728a88b2374867a4f43c817e80f21b to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/b5728a88b2374867a4f43c817e80f21b 2023-04-19 20:56:30,406 DEBUG [StoreCloser-TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/c2c5846bd82740d0ac3d4ad8028faf3f to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/c2c5846bd82740d0ac3d4ad8028faf3f 2023-04-19 20:56:30,407 DEBUG [StoreCloser-TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/579c2d90b9a14427aa382495e3b7e88c to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/579c2d90b9a14427aa382495e3b7e88c 2023-04-19 20:56:30,407 DEBUG [StoreCloser-TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/11b3f4bb7946432ca51e9ceaa3667218 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/11b3f4bb7946432ca51e9ceaa3667218 2023-04-19 20:56:30,408 DEBUG [StoreCloser-TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/90e84052bff54e508728bfe237e575ed to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/90e84052bff54e508728bfe237e575ed 2023-04-19 20:56:30,409 DEBUG [StoreCloser-TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/37ab81a43c16427e877a21033f390d02 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/37ab81a43c16427e877a21033f390d02 2023-04-19 20:56:30,410 DEBUG [StoreCloser-TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/151c342063d7435c960dcd3d8eb1c2a5 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/151c342063d7435c960dcd3d8eb1c2a5 2023-04-19 20:56:30,410 DEBUG [StoreCloser-TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/0c87feb59d8e4991ae7f1290e16b1c96 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/0c87feb59d8e4991ae7f1290e16b1c96 2023-04-19 20:56:30,411 DEBUG [StoreCloser-TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/d04fedd90da34bb2a8a3d5a76415b76f to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/d04fedd90da34bb2a8a3d5a76415b76f 2023-04-19 20:56:30,412 DEBUG [StoreCloser-TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/60d5476aad5f4856a5ec1f39d2188a1e to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/60d5476aad5f4856a5ec1f39d2188a1e 2023-04-19 20:56:30,413 DEBUG [StoreCloser-TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/ed7f3b4bb0d84c5cabb2c08befee59fc to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/ed7f3b4bb0d84c5cabb2c08befee59fc 2023-04-19 20:56:30,413 DEBUG [StoreCloser-TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/aee3852a71d64a3f9b9771d3f537012c to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/aee3852a71d64a3f9b9771d3f537012c 2023-04-19 20:56:30,414 DEBUG [StoreCloser-TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/a701b2164253433fbe6d810232758682 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/a701b2164253433fbe6d810232758682 2023-04-19 20:56:30,415 DEBUG [StoreCloser-TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/d37a0c52167b4628947ff053c9ea60cc to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/d37a0c52167b4628947ff053c9ea60cc 2023-04-19 20:56:30,416 DEBUG [StoreCloser-TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/6ca70a52f4a7454b9a4ce9efbc5a0142 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/6ca70a52f4a7454b9a4ce9efbc5a0142 2023-04-19 20:56:30,416 DEBUG [StoreCloser-TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/3ad416fa93f84a8d88acdf508a8c63f6 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/3ad416fa93f84a8d88acdf508a8c63f6 2023-04-19 20:56:30,417 DEBUG [StoreCloser-TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/db1a88079b4c495a8a7aaf0dc0430983 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/db1a88079b4c495a8a7aaf0dc0430983 2023-04-19 20:56:30,420 DEBUG [StoreCloser-TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/623b3d58bf8d4446897325a8d91870be, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/74763615523f4ee895b7c97ef70298f5, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/d3ae7695e34940de99097e297a7035ae, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/f268e27fcfa1457499b076a31a8d3e16, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/d0eeacf0aa3f452b912d0c4cf3a29642, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/2d48a2781f9b434688b5d141f60c78a2, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/9e60c267bda7478ba55d83e91c3da20b, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/9acfcde5fd774be59026248e71165777, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/a944e8d265374600957b235c230ff704, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/83399b14dae74fc2a37a1b5c1f80e84e, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/eb215aa8e54e4924910d390c3e57fc28, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/944cfc07c5b746fc98b8c2af008c2732, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/12ac88195ff64134a627c06b25f6658c, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/3e1978e7f4e945e9add84a619d00c7c2, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/279d74c15cae4c389475c0e292e93d65, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/aa28967d3bcc4f2ab972571953993c0f, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/0988ed0ae02249b6a440b2ae5400dea1, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/5278cdc7792540ae9f91bb8cbbc8a1c6, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/cf6e538d539e4db8b45d1e38aef27620, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/6b7e2a62e336464d9940c09feb96e14f, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/25eec6e070474d5fb14e4fcd26f4b4c3, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/9750333ee29047d2af6c880445330aeb, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/7693a11c4fa743a5a26c8726ef23af9c, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/33dc379f125042e7aa81de61a62d3f05, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/ec7bd190818e4214b6d960159073885b, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/a7e715205f8249a7a64057eefe5d3f47, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/eaf7d535a1fa4869ba67786952e0b4d1, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/eee3503502504cb08b4a03de1760cd79, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/9eaaa7d1ca274c85bbc7d332c7f43a0e, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/bc443c74b06c4928986c3bf33c945127, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/f48c5241a55842d6812e8404b9e4c5b2, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/3b64653e9af54c37a7f3ac49f59f3c54, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/12386e413b51429f85b6b262ee1b5e82, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/9efea23d519e445a9b3a894bdb478948, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/786395e45605458fb4c6447e2220cdd2, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/630dfd12107742b2a23a7044ee8e0d6e, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/21ca76f476b94c1ab02eb19695831e22, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/bc60fa64bc764e56bff1892182c035fd, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/85e2b51df81e4fecadd6ab1b19338e36, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/08945219adc64bb4923f4ff735a3e709, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/f2a3025aff7b4a5d9c309a18448868af] to archive 2023-04-19 20:56:30,421 DEBUG [StoreCloser-TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-19 20:56:30,422 DEBUG [StoreCloser-TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/623b3d58bf8d4446897325a8d91870be to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/623b3d58bf8d4446897325a8d91870be 2023-04-19 20:56:30,423 DEBUG [StoreCloser-TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/74763615523f4ee895b7c97ef70298f5 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/74763615523f4ee895b7c97ef70298f5 2023-04-19 20:56:30,423 DEBUG [StoreCloser-TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/d3ae7695e34940de99097e297a7035ae to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/d3ae7695e34940de99097e297a7035ae 2023-04-19 20:56:30,424 DEBUG [StoreCloser-TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/f268e27fcfa1457499b076a31a8d3e16 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/f268e27fcfa1457499b076a31a8d3e16 2023-04-19 20:56:30,425 DEBUG [StoreCloser-TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/d0eeacf0aa3f452b912d0c4cf3a29642 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/d0eeacf0aa3f452b912d0c4cf3a29642 2023-04-19 20:56:30,426 DEBUG [StoreCloser-TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/2d48a2781f9b434688b5d141f60c78a2 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/2d48a2781f9b434688b5d141f60c78a2 2023-04-19 20:56:30,427 DEBUG [StoreCloser-TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/9e60c267bda7478ba55d83e91c3da20b to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/9e60c267bda7478ba55d83e91c3da20b 2023-04-19 20:56:30,428 DEBUG [StoreCloser-TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/9acfcde5fd774be59026248e71165777 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/9acfcde5fd774be59026248e71165777 2023-04-19 20:56:30,428 DEBUG [StoreCloser-TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/a944e8d265374600957b235c230ff704 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/a944e8d265374600957b235c230ff704 2023-04-19 20:56:30,429 DEBUG [StoreCloser-TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/83399b14dae74fc2a37a1b5c1f80e84e to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/83399b14dae74fc2a37a1b5c1f80e84e 2023-04-19 20:56:30,430 DEBUG [StoreCloser-TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/eb215aa8e54e4924910d390c3e57fc28 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/eb215aa8e54e4924910d390c3e57fc28 2023-04-19 20:56:30,431 DEBUG [StoreCloser-TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/944cfc07c5b746fc98b8c2af008c2732 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/944cfc07c5b746fc98b8c2af008c2732 2023-04-19 20:56:30,432 DEBUG [StoreCloser-TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/12ac88195ff64134a627c06b25f6658c to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/12ac88195ff64134a627c06b25f6658c 2023-04-19 20:56:30,433 DEBUG [StoreCloser-TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/3e1978e7f4e945e9add84a619d00c7c2 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/3e1978e7f4e945e9add84a619d00c7c2 2023-04-19 20:56:30,434 DEBUG [StoreCloser-TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/279d74c15cae4c389475c0e292e93d65 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/279d74c15cae4c389475c0e292e93d65 2023-04-19 20:56:30,435 DEBUG [StoreCloser-TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/aa28967d3bcc4f2ab972571953993c0f to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/aa28967d3bcc4f2ab972571953993c0f 2023-04-19 20:56:30,436 DEBUG [StoreCloser-TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/0988ed0ae02249b6a440b2ae5400dea1 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/0988ed0ae02249b6a440b2ae5400dea1 2023-04-19 20:56:30,437 DEBUG [StoreCloser-TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/5278cdc7792540ae9f91bb8cbbc8a1c6 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/5278cdc7792540ae9f91bb8cbbc8a1c6 2023-04-19 20:56:30,438 DEBUG [StoreCloser-TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/cf6e538d539e4db8b45d1e38aef27620 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/cf6e538d539e4db8b45d1e38aef27620 2023-04-19 20:56:30,439 DEBUG [StoreCloser-TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/6b7e2a62e336464d9940c09feb96e14f to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/6b7e2a62e336464d9940c09feb96e14f 2023-04-19 20:56:30,439 DEBUG [StoreCloser-TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/25eec6e070474d5fb14e4fcd26f4b4c3 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/25eec6e070474d5fb14e4fcd26f4b4c3 2023-04-19 20:56:30,441 DEBUG [StoreCloser-TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/9750333ee29047d2af6c880445330aeb to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/9750333ee29047d2af6c880445330aeb 2023-04-19 20:56:30,442 DEBUG [StoreCloser-TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/7693a11c4fa743a5a26c8726ef23af9c to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/7693a11c4fa743a5a26c8726ef23af9c 2023-04-19 20:56:30,443 DEBUG [StoreCloser-TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/33dc379f125042e7aa81de61a62d3f05 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/33dc379f125042e7aa81de61a62d3f05 2023-04-19 20:56:30,444 DEBUG [StoreCloser-TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/ec7bd190818e4214b6d960159073885b to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/ec7bd190818e4214b6d960159073885b 2023-04-19 20:56:30,445 DEBUG [StoreCloser-TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/a7e715205f8249a7a64057eefe5d3f47 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/a7e715205f8249a7a64057eefe5d3f47 2023-04-19 20:56:30,447 DEBUG [StoreCloser-TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/eaf7d535a1fa4869ba67786952e0b4d1 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/eaf7d535a1fa4869ba67786952e0b4d1 2023-04-19 20:56:30,448 DEBUG [StoreCloser-TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/eee3503502504cb08b4a03de1760cd79 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/eee3503502504cb08b4a03de1760cd79 2023-04-19 20:56:30,449 DEBUG [StoreCloser-TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/9eaaa7d1ca274c85bbc7d332c7f43a0e to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/9eaaa7d1ca274c85bbc7d332c7f43a0e 2023-04-19 20:56:30,450 DEBUG [StoreCloser-TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/bc443c74b06c4928986c3bf33c945127 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/bc443c74b06c4928986c3bf33c945127 2023-04-19 20:56:30,451 DEBUG [StoreCloser-TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/f48c5241a55842d6812e8404b9e4c5b2 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/f48c5241a55842d6812e8404b9e4c5b2 2023-04-19 20:56:30,452 DEBUG [StoreCloser-TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/3b64653e9af54c37a7f3ac49f59f3c54 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/3b64653e9af54c37a7f3ac49f59f3c54 2023-04-19 20:56:30,453 DEBUG [StoreCloser-TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/12386e413b51429f85b6b262ee1b5e82 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/12386e413b51429f85b6b262ee1b5e82 2023-04-19 20:56:30,454 DEBUG [StoreCloser-TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/9efea23d519e445a9b3a894bdb478948 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/9efea23d519e445a9b3a894bdb478948 2023-04-19 20:56:30,455 DEBUG [StoreCloser-TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/786395e45605458fb4c6447e2220cdd2 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/786395e45605458fb4c6447e2220cdd2 2023-04-19 20:56:30,456 DEBUG [StoreCloser-TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/630dfd12107742b2a23a7044ee8e0d6e to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/630dfd12107742b2a23a7044ee8e0d6e 2023-04-19 20:56:30,457 DEBUG [StoreCloser-TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/21ca76f476b94c1ab02eb19695831e22 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/21ca76f476b94c1ab02eb19695831e22 2023-04-19 20:56:30,458 DEBUG [StoreCloser-TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/bc60fa64bc764e56bff1892182c035fd to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/bc60fa64bc764e56bff1892182c035fd 2023-04-19 20:56:30,459 DEBUG [StoreCloser-TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/85e2b51df81e4fecadd6ab1b19338e36 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/85e2b51df81e4fecadd6ab1b19338e36 2023-04-19 20:56:30,460 DEBUG [StoreCloser-TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/08945219adc64bb4923f4ff735a3e709 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/08945219adc64bb4923f4ff735a3e709 2023-04-19 20:56:30,461 DEBUG [StoreCloser-TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/f2a3025aff7b4a5d9c309a18448868af to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/f2a3025aff7b4a5d9c309a18448868af 2023-04-19 20:56:30,465 DEBUG [StoreCloser-TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/e75d282e6e154426a1c0a12b4be75bd7, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/a79628ffa4934ee4afb6fde94c029cd0, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/459f086d2f8a4a988a5447fbb85d6d74, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/31d9e9c62f0242e8bf31e58e758240f1, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/ec555bd2579447849138ddb079eeef00, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/66ec2c0da0ab4ddfb833b817a8649e63, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/5c6e274922f44ba7aa8a69aced8334cb, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/a5d417ff25964628a8b91bef78c15d8b, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/c55e6ce76d53470f9a91643a0fa72029, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/e3fddbd147a44fb89c13661b67f3f832, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/17dd83e3da5247d9b0da715ea48bda24, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/3fcf93729c9f44ebbb082fda90a8d510, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/4098d024bfea43c49736438980462d39, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/972d94d633ce4e0185cd4ba3093e59ec, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/cc4cecf9f98d402081d04108ed1a6524, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/410bffe0cea7475383bc553c8a8923ed, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/a3fd7c0f1b884fa79ddb4b4169951bc5, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/0b52a0554f824d179867e0bcf5ff4ecf, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/caf2f2e21ba949b2bbd08822eff3eb91, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/baa0548f46a846b4b48e1a25e7f3913b, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/df3c8da2747646e589d8c282d19e756b, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/826a0a4407cb488db170becc2217b51c, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/7dabfe6513b64939b4d64f5432d2a4dc, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/e3c72c0402434b0d999f0f6483a108d9, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/4f6b7a4ac90844378013533328745a19, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/11f608a8f60f486a8094a7304764ae55, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/fa95bf53b19645f29d784ed3d3a5632e, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/0f6ae49ecfe74a13bb4bc05706a01e7d, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/11379df5f405422ab142bf9546f8863d, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/3b0a0c2531804a32bb02046673d6ba7e, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/ea89bf0f7e3c4e87857c7bb120c24c77, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/aa108859b8444a58bc53bfe600264938, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/f1a7cebb8d78421081faf72f3fef352e, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/2b4460932e38440099225f3c7589f7c1, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/ae53b4179b724aa888e021e60fb3c784, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/b1081035997f4838af62a1c177db5427, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/37ef51f107644be5b36e45327b231d10, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/9f499bfde60447bb80a012903f69fe0d, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/255f7a2e44bb4d44bef6296297a953fe, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/658f4875e9e64d7c87638411cc43601c] to archive 2023-04-19 20:56:30,466 DEBUG [StoreCloser-TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-19 20:56:30,467 DEBUG [StoreCloser-TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/e75d282e6e154426a1c0a12b4be75bd7 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/e75d282e6e154426a1c0a12b4be75bd7 2023-04-19 20:56:30,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-19 20:56:30,468 DEBUG [StoreCloser-TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/a79628ffa4934ee4afb6fde94c029cd0 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/a79628ffa4934ee4afb6fde94c029cd0 2023-04-19 20:56:30,469 DEBUG [StoreCloser-TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/459f086d2f8a4a988a5447fbb85d6d74 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/459f086d2f8a4a988a5447fbb85d6d74 2023-04-19 20:56:30,470 DEBUG [StoreCloser-TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/31d9e9c62f0242e8bf31e58e758240f1 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/31d9e9c62f0242e8bf31e58e758240f1 2023-04-19 20:56:30,471 DEBUG [StoreCloser-TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/ec555bd2579447849138ddb079eeef00 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/ec555bd2579447849138ddb079eeef00 2023-04-19 20:56:30,472 DEBUG [StoreCloser-TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/66ec2c0da0ab4ddfb833b817a8649e63 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/66ec2c0da0ab4ddfb833b817a8649e63 2023-04-19 20:56:30,473 DEBUG [StoreCloser-TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/5c6e274922f44ba7aa8a69aced8334cb to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/5c6e274922f44ba7aa8a69aced8334cb 2023-04-19 20:56:30,474 DEBUG [StoreCloser-TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/a5d417ff25964628a8b91bef78c15d8b to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/a5d417ff25964628a8b91bef78c15d8b 2023-04-19 20:56:30,475 DEBUG [StoreCloser-TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/c55e6ce76d53470f9a91643a0fa72029 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/c55e6ce76d53470f9a91643a0fa72029 2023-04-19 20:56:30,476 DEBUG [StoreCloser-TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/e3fddbd147a44fb89c13661b67f3f832 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/e3fddbd147a44fb89c13661b67f3f832 2023-04-19 20:56:30,477 DEBUG [StoreCloser-TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/17dd83e3da5247d9b0da715ea48bda24 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/17dd83e3da5247d9b0da715ea48bda24 2023-04-19 20:56:30,478 DEBUG [StoreCloser-TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/3fcf93729c9f44ebbb082fda90a8d510 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/3fcf93729c9f44ebbb082fda90a8d510 2023-04-19 20:56:30,479 DEBUG [StoreCloser-TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/4098d024bfea43c49736438980462d39 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/4098d024bfea43c49736438980462d39 2023-04-19 20:56:30,480 DEBUG [StoreCloser-TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/972d94d633ce4e0185cd4ba3093e59ec to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/972d94d633ce4e0185cd4ba3093e59ec 2023-04-19 20:56:30,481 DEBUG [StoreCloser-TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/cc4cecf9f98d402081d04108ed1a6524 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/cc4cecf9f98d402081d04108ed1a6524 2023-04-19 20:56:30,482 DEBUG [StoreCloser-TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/410bffe0cea7475383bc553c8a8923ed to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/410bffe0cea7475383bc553c8a8923ed 2023-04-19 20:56:30,482 DEBUG [StoreCloser-TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/a3fd7c0f1b884fa79ddb4b4169951bc5 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/a3fd7c0f1b884fa79ddb4b4169951bc5 2023-04-19 20:56:30,483 DEBUG [StoreCloser-TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/0b52a0554f824d179867e0bcf5ff4ecf to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/0b52a0554f824d179867e0bcf5ff4ecf 2023-04-19 20:56:30,484 DEBUG [StoreCloser-TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/caf2f2e21ba949b2bbd08822eff3eb91 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/caf2f2e21ba949b2bbd08822eff3eb91 2023-04-19 20:56:30,485 DEBUG [StoreCloser-TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/baa0548f46a846b4b48e1a25e7f3913b to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/baa0548f46a846b4b48e1a25e7f3913b 2023-04-19 20:56:30,486 DEBUG [StoreCloser-TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/df3c8da2747646e589d8c282d19e756b to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/df3c8da2747646e589d8c282d19e756b 2023-04-19 20:56:30,487 DEBUG [StoreCloser-TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/826a0a4407cb488db170becc2217b51c to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/826a0a4407cb488db170becc2217b51c 2023-04-19 20:56:30,488 DEBUG [StoreCloser-TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/7dabfe6513b64939b4d64f5432d2a4dc to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/7dabfe6513b64939b4d64f5432d2a4dc 2023-04-19 20:56:30,489 DEBUG [StoreCloser-TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/e3c72c0402434b0d999f0f6483a108d9 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/e3c72c0402434b0d999f0f6483a108d9 2023-04-19 20:56:30,490 DEBUG [StoreCloser-TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/4f6b7a4ac90844378013533328745a19 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/4f6b7a4ac90844378013533328745a19 2023-04-19 20:56:30,492 DEBUG [StoreCloser-TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/11f608a8f60f486a8094a7304764ae55 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/11f608a8f60f486a8094a7304764ae55 2023-04-19 20:56:30,493 DEBUG [StoreCloser-TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/fa95bf53b19645f29d784ed3d3a5632e to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/fa95bf53b19645f29d784ed3d3a5632e 2023-04-19 20:56:30,494 DEBUG [StoreCloser-TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/0f6ae49ecfe74a13bb4bc05706a01e7d to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/0f6ae49ecfe74a13bb4bc05706a01e7d 2023-04-19 20:56:30,495 DEBUG [StoreCloser-TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/11379df5f405422ab142bf9546f8863d to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/11379df5f405422ab142bf9546f8863d 2023-04-19 20:56:30,497 DEBUG [StoreCloser-TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/3b0a0c2531804a32bb02046673d6ba7e to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/3b0a0c2531804a32bb02046673d6ba7e 2023-04-19 20:56:30,498 DEBUG [StoreCloser-TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/ea89bf0f7e3c4e87857c7bb120c24c77 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/ea89bf0f7e3c4e87857c7bb120c24c77 2023-04-19 20:56:30,499 DEBUG [StoreCloser-TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/aa108859b8444a58bc53bfe600264938 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/aa108859b8444a58bc53bfe600264938 2023-04-19 20:56:30,500 DEBUG [StoreCloser-TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/f1a7cebb8d78421081faf72f3fef352e to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/f1a7cebb8d78421081faf72f3fef352e 2023-04-19 20:56:30,501 DEBUG [StoreCloser-TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/2b4460932e38440099225f3c7589f7c1 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/2b4460932e38440099225f3c7589f7c1 2023-04-19 20:56:30,501 DEBUG [StoreCloser-TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/ae53b4179b724aa888e021e60fb3c784 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/ae53b4179b724aa888e021e60fb3c784 2023-04-19 20:56:30,502 DEBUG [StoreCloser-TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/b1081035997f4838af62a1c177db5427 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/b1081035997f4838af62a1c177db5427 2023-04-19 20:56:30,503 DEBUG [StoreCloser-TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/37ef51f107644be5b36e45327b231d10 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/37ef51f107644be5b36e45327b231d10 2023-04-19 20:56:30,504 DEBUG [StoreCloser-TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/9f499bfde60447bb80a012903f69fe0d to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/9f499bfde60447bb80a012903f69fe0d 2023-04-19 20:56:30,505 DEBUG [StoreCloser-TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/255f7a2e44bb4d44bef6296297a953fe to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/255f7a2e44bb4d44bef6296297a953fe 2023-04-19 20:56:30,506 DEBUG [StoreCloser-TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/658f4875e9e64d7c87638411cc43601c to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/658f4875e9e64d7c87638411cc43601c 2023-04-19 20:56:30,510 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/recovered.edits/728.seqid, newMaxSeqId=728, maxSeqId=4 2023-04-19 20:56:30,511 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267. 2023-04-19 20:56:30,511 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 6873ba93789e63dd9f14391208147267: 2023-04-19 20:56:30,512 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 6873ba93789e63dd9f14391208147267 2023-04-19 20:56:30,512 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=63 updating hbase:meta row=6873ba93789e63dd9f14391208147267, regionState=CLOSED 2023-04-19 20:56:30,513 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681937790512"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681937790512"}]},"ts":"1681937790512"} 2023-04-19 20:56:30,514 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=64, resume processing ppid=63 2023-04-19 20:56:30,514 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=64, ppid=63, state=SUCCESS; CloseRegionProcedure 6873ba93789e63dd9f14391208147267, server=jenkins-hbase4.apache.org,35247,1681937616971 in 1.1440 sec 2023-04-19 20:56:30,516 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=63, resume processing ppid=62 2023-04-19 20:56:30,516 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=63, ppid=62, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=6873ba93789e63dd9f14391208147267, UNASSIGN in 1.1480 sec 2023-04-19 20:56:30,516 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681937790516"}]},"ts":"1681937790516"} 2023-04-19 20:56:30,517 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-19 20:56:30,520 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-19 20:56:30,521 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=62, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.1590 sec 2023-04-19 20:56:31,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-19 20:56:31,468 INFO [Listener at localhost/34893] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 62 completed 2023-04-19 20:56:31,469 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-19 20:56:31,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] procedure2.ProcedureExecutor(1029): Stored pid=65, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-19 20:56:31,471 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=65, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-19 20:56:31,472 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=65, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-19 20:56:31,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1227): Checking to see if procedure is done pid=65 2023-04-19 20:56:31,477 DEBUG [HFileArchiver-13] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/.tmp/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267 2023-04-19 20:56:31,478 DEBUG [HFileArchiver-13] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/.tmp/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A, FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/.tmp/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B, FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/.tmp/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C, FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/.tmp/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/recovered.edits] 2023-04-19 20:56:31,481 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/.tmp/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/eb4d9bb777fd4deaa3e44a1e77607fcb to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/eb4d9bb777fd4deaa3e44a1e77607fcb 2023-04-19 20:56:31,482 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/.tmp/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/f6368fc7393d40f883e4513f67d53cb9 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/A/f6368fc7393d40f883e4513f67d53cb9 2023-04-19 20:56:31,484 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/.tmp/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/e0c6e1e4e1b24ee6b88863dc4f208b17 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/e0c6e1e4e1b24ee6b88863dc4f208b17 2023-04-19 20:56:31,485 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/.tmp/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/fc97ee9a48494d448a63b5d093856aa9 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/B/fc97ee9a48494d448a63b5d093856aa9 2023-04-19 20:56:31,486 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/.tmp/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/0f0730f31db44bf6bbcadb132cc01a31 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/0f0730f31db44bf6bbcadb132cc01a31 2023-04-19 20:56:31,487 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/.tmp/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/d57f9f95bf8647c0a4bef1f7d0b18692 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/C/d57f9f95bf8647c0a4bef1f7d0b18692 2023-04-19 20:56:31,489 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/.tmp/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/recovered.edits/728.seqid to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267/recovered.edits/728.seqid 2023-04-19 20:56:31,490 DEBUG [HFileArchiver-13] backup.HFileArchiver(596): Deleted hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/.tmp/data/default/TestAcidGuarantees/6873ba93789e63dd9f14391208147267 2023-04-19 20:56:31,490 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-19 20:56:31,490 DEBUG [PEWorker-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-19 20:56:31,491 DEBUG [PEWorker-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-04-19 20:56:31,493 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304191520ea61b5964207b939a9b006f9f0c3 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304191520ea61b5964207b939a9b006f9f0c3 2023-04-19 20:56:31,494 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304191bb16a345dfe4f1c8c83892e941a7f99 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304191bb16a345dfe4f1c8c83892e941a7f99 2023-04-19 20:56:31,495 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419284fa4e20d244846ba2d5a60871ac3cc to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419284fa4e20d244846ba2d5a60871ac3cc 2023-04-19 20:56:31,496 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041928dc060a636540868b0f5f486d7cb5f9 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041928dc060a636540868b0f5f486d7cb5f9 2023-04-19 20:56:31,497 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304192a49aa7926c44f15a2fcd3232a42781f to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304192a49aa7926c44f15a2fcd3232a42781f 2023-04-19 20:56:31,498 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304192fff06512e764eeb8e6e2e2f6c39e2a0 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304192fff06512e764eeb8e6e2e2f6c39e2a0 2023-04-19 20:56:31,498 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041932c84ed833d1437688fa355654a3b166 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041932c84ed833d1437688fa355654a3b166 2023-04-19 20:56:31,499 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304193ab4a47a6d8740719eff0f5fb9ec78ff to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304193ab4a47a6d8740719eff0f5fb9ec78ff 2023-04-19 20:56:31,500 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304194ad1ea35507c4845966484b15526bb3e to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304194ad1ea35507c4845966484b15526bb3e 2023-04-19 20:56:31,501 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304194b72c7d0b4224bb48b607d3bdad86621 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304194b72c7d0b4224bb48b607d3bdad86621 2023-04-19 20:56:31,502 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041951c6ec6a80374a51a0d107266ade06a6 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041951c6ec6a80374a51a0d107266ade06a6 2023-04-19 20:56:31,503 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304195ae8e0bd317f436e8dae5510adb191a4 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304195ae8e0bd317f436e8dae5510adb191a4 2023-04-19 20:56:31,504 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304195c6c9e2130714c219c21147c92d97a32 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304195c6c9e2130714c219c21147c92d97a32 2023-04-19 20:56:31,504 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304195e9931f9c02248b4b523ad3370cfc4c0 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304195e9931f9c02248b4b523ad3370cfc4c0 2023-04-19 20:56:31,505 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304195ec0d5c5608d43a7a9c3e5cbbf3e5335 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304195ec0d5c5608d43a7a9c3e5cbbf3e5335 2023-04-19 20:56:31,506 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419635520dbce3844d4949094eaf511b5a5 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419635520dbce3844d4949094eaf511b5a5 2023-04-19 20:56:31,507 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419635ad6cc96244b768208a0cce991a482 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419635ad6cc96244b768208a0cce991a482 2023-04-19 20:56:31,508 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041965b3241125eb4616a370b96cb35a202a to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041965b3241125eb4616a370b96cb35a202a 2023-04-19 20:56:31,509 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041967ecd5fecadf48928395ed882565e32d to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041967ecd5fecadf48928395ed882565e32d 2023-04-19 20:56:31,509 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419789f271c201d4003af84f5f78c11bf4d to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419789f271c201d4003af84f5f78c11bf4d 2023-04-19 20:56:31,510 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419829a92fb9a124004b0d08db3a66f12c6 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419829a92fb9a124004b0d08db3a66f12c6 2023-04-19 20:56:31,511 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041989499b73be374f819684e66b46a3595d to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041989499b73be374f819684e66b46a3595d 2023-04-19 20:56:31,512 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304198fd3f56c29ac4920b120d85396963baf to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304198fd3f56c29ac4920b120d85396963baf 2023-04-19 20:56:31,512 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304199cb9fc35d1f640d1a908890300190c85 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304199cb9fc35d1f640d1a908890300190c85 2023-04-19 20:56:31,513 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419a45448a74b0541fa9565250d0b297c33 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419a45448a74b0541fa9565250d0b297c33 2023-04-19 20:56:31,514 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419ac03677948bd4dda91289b467ce91eb9 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419ac03677948bd4dda91289b467ce91eb9 2023-04-19 20:56:31,515 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419ad27124eaf3849e18338f58fda8fd303 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419ad27124eaf3849e18338f58fda8fd303 2023-04-19 20:56:31,515 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419bf3471879a724a238b1cd0c48e33ab0a to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419bf3471879a724a238b1cd0c48e33ab0a 2023-04-19 20:56:31,516 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419c942bf57f1974c0c948bc10c2e39e754 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419c942bf57f1974c0c948bc10c2e39e754 2023-04-19 20:56:31,517 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419e62b7ddf153b413dbf7a444d0ed8bc03 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419e62b7ddf153b413dbf7a444d0ed8bc03 2023-04-19 20:56:31,518 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419ecc0ceb8d0b743d0b126137582502c6a to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419ecc0ceb8d0b743d0b126137582502c6a 2023-04-19 20:56:31,518 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419f042715ab0054a799a40c97a0ce69614 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419f042715ab0054a799a40c97a0ce69614 2023-04-19 20:56:31,519 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419f500c071ce954f7abccf50c691ec2a57 to hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419f500c071ce954f7abccf50c691ec2a57 2023-04-19 20:56:31,519 DEBUG [PEWorker-5] backup.HFileArchiver(596): Deleted hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-19 20:56:31,521 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=65, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-19 20:56:31,522 WARN [PEWorker-5] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-19 20:56:31,524 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-19 20:56:31,525 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=65, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-19 20:56:31,525 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-19 20:56:31,525 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1681937791525"}]},"ts":"9223372036854775807"} 2023-04-19 20:56:31,526 INFO [PEWorker-5] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-19 20:56:31,526 DEBUG [PEWorker-5] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 6873ba93789e63dd9f14391208147267, NAME => 'TestAcidGuarantees,,1681937764358.6873ba93789e63dd9f14391208147267.', STARTKEY => '', ENDKEY => ''}] 2023-04-19 20:56:31,526 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-19 20:56:31,526 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1681937791526"}]},"ts":"9223372036854775807"} 2023-04-19 20:56:31,527 INFO [PEWorker-5] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-19 20:56:31,529 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(130): Finished pid=65, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-19 20:56:31,529 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=65, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 60 msec 2023-04-19 20:56:31,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37181] master.MasterRpcServices(1227): Checking to see if procedure is done pid=65 2023-04-19 20:56:31,573 INFO [Listener at localhost/34893] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 65 completed 2023-04-19 20:56:31,586 INFO [Listener at localhost/34893] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobGetAtomicity Thread=229 (was 230), OpenFileDescriptor=596 (was 612), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=442 (was 415) - SystemLoadAverage LEAK? -, ProcessCount=172 (was 172), AvailableMemoryMB=5723 (was 5649) - AvailableMemoryMB LEAK? - 2023-04-19 20:56:31,587 INFO [Listener at localhost/34893] hbase.HBaseTestingUtility(1286): Shutting down minicluster 2023-04-19 20:56:31,587 INFO [Listener at localhost/34893] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-19 20:56:31,587 DEBUG [Listener at localhost/34893] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x461317a6 to 127.0.0.1:58736 2023-04-19 20:56:31,587 DEBUG [Listener at localhost/34893] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-19 20:56:31,587 DEBUG [Listener at localhost/34893] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-04-19 20:56:31,588 DEBUG [Listener at localhost/34893] util.JVMClusterUtil(257): Found active master hash=1674471517, stopped=false 2023-04-19 20:56:31,588 INFO [Listener at localhost/34893] master.ServerManager(901): Cluster shutdown requested of master=jenkins-hbase4.apache.org,37181,1681937615733 2023-04-19 20:56:31,589 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-04-19 20:56:31,589 INFO [Listener at localhost/34893] procedure2.ProcedureExecutor(629): Stopping 2023-04-19 20:56:31,589 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-04-19 20:56:31,589 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-19 20:56:31,589 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-19 20:56:31,589 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-19 20:56:31,590 DEBUG [Listener at localhost/34893] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2c57173e to 127.0.0.1:58736 2023-04-19 20:56:31,590 DEBUG [Listener at localhost/34893] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-19 20:56:31,590 INFO [Listener at localhost/34893] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,35247,1681937616971' ***** 2023-04-19 20:56:31,590 INFO [Listener at localhost/34893] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-04-19 20:56:31,590 INFO [RS:0;jenkins-hbase4:35247] regionserver.HeapMemoryManager(220): Stopping 2023-04-19 20:56:31,590 INFO [RS:0;jenkins-hbase4:35247] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-04-19 20:56:31,591 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-04-19 20:56:31,591 INFO [RS:0;jenkins-hbase4:35247] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-04-19 20:56:31,591 INFO [RS:0;jenkins-hbase4:35247] regionserver.HRegionServer(3304): Received CLOSE for e22e0fa16aeb37f0afb0679ab2e27008 2023-04-19 20:56:31,592 INFO [RS:0;jenkins-hbase4:35247] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:31,592 DEBUG [RS:0;jenkins-hbase4:35247] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x735e461f to 127.0.0.1:58736 2023-04-19 20:56:31,592 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing e22e0fa16aeb37f0afb0679ab2e27008, disabling compactions & flushes 2023-04-19 20:56:31,592 DEBUG [RS:0;jenkins-hbase4:35247] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-19 20:56:31,592 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1681937619300.e22e0fa16aeb37f0afb0679ab2e27008. 2023-04-19 20:56:31,592 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1681937619300.e22e0fa16aeb37f0afb0679ab2e27008. 2023-04-19 20:56:31,592 INFO [RS:0;jenkins-hbase4:35247] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-04-19 20:56:31,592 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1681937619300.e22e0fa16aeb37f0afb0679ab2e27008. after waiting 0 ms 2023-04-19 20:56:31,592 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1681937619300.e22e0fa16aeb37f0afb0679ab2e27008. 2023-04-19 20:56:31,592 INFO [RS:0;jenkins-hbase4:35247] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-04-19 20:56:31,592 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing e22e0fa16aeb37f0afb0679ab2e27008 1/1 column families, dataSize=78 B heapSize=488 B 2023-04-19 20:56:31,592 INFO [RS:0;jenkins-hbase4:35247] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-04-19 20:56:31,592 INFO [RS:0;jenkins-hbase4:35247] regionserver.HRegionServer(3304): Received CLOSE for 1588230740 2023-04-19 20:56:31,593 INFO [RS:0;jenkins-hbase4:35247] regionserver.HRegionServer(1475): Waiting on 2 regions to close 2023-04-19 20:56:31,593 DEBUG [RS:0;jenkins-hbase4:35247] regionserver.HRegionServer(1479): Online Regions={e22e0fa16aeb37f0afb0679ab2e27008=hbase:namespace,,1681937619300.e22e0fa16aeb37f0afb0679ab2e27008., 1588230740=hbase:meta,,1.1588230740} 2023-04-19 20:56:31,593 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-04-19 20:56:31,593 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-04-19 20:56:31,593 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-04-19 20:56:31,593 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-04-19 20:56:31,593 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-04-19 20:56:31,593 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=20.92 KB heapSize=36.24 KB 2023-04-19 20:56:31,595 DEBUG [RS:0;jenkins-hbase4:35247] regionserver.HRegionServer(1505): Waiting on 1588230740, e22e0fa16aeb37f0afb0679ab2e27008 2023-04-19 20:56:31,606 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=18.26 KB at sequenceid=93 (bloomFilter=false), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/hbase/meta/1588230740/.tmp/info/4da12f225836465eb4cc1f56bc432d40 2023-04-19 20:56:31,610 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 4da12f225836465eb4cc1f56bc432d40 2023-04-19 20:56:31,618 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=588 B at sequenceid=93 (bloomFilter=false), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/hbase/meta/1588230740/.tmp/rep_barrier/2d079f0c0402470eaa941eee0472c66e 2023-04-19 20:56:31,625 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 2d079f0c0402470eaa941eee0472c66e 2023-04-19 20:56:31,630 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-04-19 20:56:31,795 DEBUG [RS:0;jenkins-hbase4:35247] regionserver.HRegionServer(1505): Waiting on 1588230740, e22e0fa16aeb37f0afb0679ab2e27008 2023-04-19 20:56:31,995 DEBUG [RS:0;jenkins-hbase4:35247] regionserver.HRegionServer(1505): Waiting on 1588230740, e22e0fa16aeb37f0afb0679ab2e27008 2023-04-19 20:56:32,000 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=78 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/hbase/namespace/e22e0fa16aeb37f0afb0679ab2e27008/.tmp/info/44f32f796e18471497da3813a14c21b2 2023-04-19 20:56:32,004 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/hbase/namespace/e22e0fa16aeb37f0afb0679ab2e27008/.tmp/info/44f32f796e18471497da3813a14c21b2 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/hbase/namespace/e22e0fa16aeb37f0afb0679ab2e27008/info/44f32f796e18471497da3813a14c21b2 2023-04-19 20:56:32,006 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/hbase/namespace/e22e0fa16aeb37f0afb0679ab2e27008/info/44f32f796e18471497da3813a14c21b2, entries=2, sequenceid=6, filesize=4.8 K 2023-04-19 20:56:32,007 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for e22e0fa16aeb37f0afb0679ab2e27008 in 415ms, sequenceid=6, compaction requested=false 2023-04-19 20:56:32,012 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/hbase/namespace/e22e0fa16aeb37f0afb0679ab2e27008/recovered.edits/9.seqid, newMaxSeqId=9, maxSeqId=1 2023-04-19 20:56:32,013 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1681937619300.e22e0fa16aeb37f0afb0679ab2e27008. 2023-04-19 20:56:32,013 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for e22e0fa16aeb37f0afb0679ab2e27008: 2023-04-19 20:56:32,013 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1681937619300.e22e0fa16aeb37f0afb0679ab2e27008. 2023-04-19 20:56:32,031 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.08 KB at sequenceid=93 (bloomFilter=false), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/hbase/meta/1588230740/.tmp/table/5c640d67722f4c5385228fb8ee7d8b01 2023-04-19 20:56:32,034 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 5c640d67722f4c5385228fb8ee7d8b01 2023-04-19 20:56:32,034 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/hbase/meta/1588230740/.tmp/info/4da12f225836465eb4cc1f56bc432d40 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/hbase/meta/1588230740/info/4da12f225836465eb4cc1f56bc432d40 2023-04-19 20:56:32,037 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 4da12f225836465eb4cc1f56bc432d40 2023-04-19 20:56:32,037 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/hbase/meta/1588230740/info/4da12f225836465eb4cc1f56bc432d40, entries=22, sequenceid=93, filesize=7.1 K 2023-04-19 20:56:32,039 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/hbase/meta/1588230740/.tmp/rep_barrier/2d079f0c0402470eaa941eee0472c66e as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/hbase/meta/1588230740/rep_barrier/2d079f0c0402470eaa941eee0472c66e 2023-04-19 20:56:32,042 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 2d079f0c0402470eaa941eee0472c66e 2023-04-19 20:56:32,042 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/hbase/meta/1588230740/rep_barrier/2d079f0c0402470eaa941eee0472c66e, entries=6, sequenceid=93, filesize=5.5 K 2023-04-19 20:56:32,042 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/hbase/meta/1588230740/.tmp/table/5c640d67722f4c5385228fb8ee7d8b01 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/hbase/meta/1588230740/table/5c640d67722f4c5385228fb8ee7d8b01 2023-04-19 20:56:32,045 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 5c640d67722f4c5385228fb8ee7d8b01 2023-04-19 20:56:32,045 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/hbase/meta/1588230740/table/5c640d67722f4c5385228fb8ee7d8b01, entries=9, sequenceid=93, filesize=5.5 K 2023-04-19 20:56:32,045 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.92 KB/21417, heapSize ~36.20 KB/37064, currentSize=0 B/0 for 1588230740 in 452ms, sequenceid=93, compaction requested=false 2023-04-19 20:56:32,052 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/data/hbase/meta/1588230740/recovered.edits/96.seqid, newMaxSeqId=96, maxSeqId=1 2023-04-19 20:56:32,053 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-04-19 20:56:32,054 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-04-19 20:56:32,054 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-04-19 20:56:32,054 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-04-19 20:56:32,196 INFO [RS:0;jenkins-hbase4:35247] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,35247,1681937616971; all regions closed. 2023-04-19 20:56:32,204 DEBUG [RS:0;jenkins-hbase4:35247] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/oldWALs 2023-04-19 20:56:32,204 INFO [RS:0;jenkins-hbase4:35247] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C35247%2C1681937616971.meta:.meta(num 1681937619027) 2023-04-19 20:56:32,208 DEBUG [RS:0;jenkins-hbase4:35247] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/oldWALs 2023-04-19 20:56:32,208 INFO [RS:0;jenkins-hbase4:35247] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C35247%2C1681937616971:(num 1681937618475) 2023-04-19 20:56:32,208 DEBUG [RS:0;jenkins-hbase4:35247] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-19 20:56:32,209 INFO [RS:0;jenkins-hbase4:35247] regionserver.LeaseManager(133): Closed leases 2023-04-19 20:56:32,209 INFO [RS:0;jenkins-hbase4:35247] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS] on shutdown 2023-04-19 20:56:32,209 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-04-19 20:56:32,211 INFO [RS:0;jenkins-hbase4:35247] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:35247 2023-04-19 20:56:32,217 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,35247,1681937616971 2023-04-19 20:56:32,217 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-04-19 20:56:32,217 ERROR [Listener at localhost/34893-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@20566bd4 rejected from java.util.concurrent.ThreadPoolExecutor@2469eb06[Shutting down, pool size = 1, active threads = 0, queued tasks = 0, completed tasks = 2461] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-19 20:56:32,218 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-04-19 20:56:32,218 ERROR [Listener at localhost/34893-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@43dbcf84 rejected from java.util.concurrent.ThreadPoolExecutor@2469eb06[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2461] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-19 20:56:32,219 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,35247,1681937616971] 2023-04-19 20:56:32,220 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,35247,1681937616971; numProcessing=1 2023-04-19 20:56:32,221 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,35247,1681937616971 already deleted, retry=false 2023-04-19 20:56:32,221 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase4.apache.org,35247,1681937616971 expired; onlineServers=0 2023-04-19 20:56:32,221 INFO [RegionServerTracker-0] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,37181,1681937615733' ***** 2023-04-19 20:56:32,221 INFO [RegionServerTracker-0] regionserver.HRegionServer(2310): STOPPED: Cluster shutdown set; onlineServer=0 2023-04-19 20:56:32,221 DEBUG [M:0;jenkins-hbase4:37181] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@22f36266, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-04-19 20:56:32,221 INFO [M:0;jenkins-hbase4:37181] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,37181,1681937615733 2023-04-19 20:56:32,221 INFO [M:0;jenkins-hbase4:37181] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,37181,1681937615733; all regions closed. 2023-04-19 20:56:32,221 DEBUG [M:0;jenkins-hbase4:37181] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-19 20:56:32,221 DEBUG [M:0;jenkins-hbase4:37181] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-04-19 20:56:32,222 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-04-19 20:56:32,222 DEBUG [M:0;jenkins-hbase4:37181] cleaner.HFileCleaner(317): Stopping file delete threads 2023-04-19 20:56:32,222 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1681937618134] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1681937618134,5,FailOnTimeoutGroup] 2023-04-19 20:56:32,222 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1681937618133] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1681937618133,5,FailOnTimeoutGroup] 2023-04-19 20:56:32,222 INFO [M:0;jenkins-hbase4:37181] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-04-19 20:56:32,222 INFO [M:0;jenkins-hbase4:37181] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-04-19 20:56:32,222 INFO [M:0;jenkins-hbase4:37181] hbase.ChoreService(369): Chore service for: master/jenkins-hbase4:0 had [] on shutdown 2023-04-19 20:56:32,222 DEBUG [M:0;jenkins-hbase4:37181] master.HMaster(1512): Stopping service threads 2023-04-19 20:56:32,223 INFO [M:0;jenkins-hbase4:37181] procedure2.RemoteProcedureDispatcher(118): Stopping procedure remote dispatcher 2023-04-19 20:56:32,223 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-04-19 20:56:32,223 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-19 20:56:32,223 ERROR [M:0;jenkins-hbase4:37181] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[HFileArchiver-11,5,PEWorkerGroup] Thread[HFileArchiver-12,5,PEWorkerGroup] Thread[IPC Client (201722471) connection to localhost/127.0.0.1:43431 from jenkins,5,PEWorkerGroup] Thread[HFileArchiver-13,5,PEWorkerGroup] 2023-04-19 20:56:32,223 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-04-19 20:56:32,224 INFO [M:0;jenkins-hbase4:37181] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-04-19 20:56:32,224 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-04-19 20:56:32,224 DEBUG [M:0;jenkins-hbase4:37181] zookeeper.ZKUtil(398): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-04-19 20:56:32,224 WARN [M:0;jenkins-hbase4:37181] master.ActiveMasterManager(326): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-04-19 20:56:32,224 INFO [M:0;jenkins-hbase4:37181] assignment.AssignmentManager(315): Stopping assignment manager 2023-04-19 20:56:32,224 INFO [M:0;jenkins-hbase4:37181] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-04-19 20:56:32,225 DEBUG [M:0;jenkins-hbase4:37181] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-04-19 20:56:32,225 INFO [M:0;jenkins-hbase4:37181] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-19 20:56:32,225 DEBUG [M:0;jenkins-hbase4:37181] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-19 20:56:32,225 DEBUG [M:0;jenkins-hbase4:37181] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-04-19 20:56:32,225 DEBUG [M:0;jenkins-hbase4:37181] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-19 20:56:32,225 INFO [M:0;jenkins-hbase4:37181] regionserver.HRegion(2745): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=344.81 KB heapSize=393.60 KB 2023-04-19 20:56:32,237 INFO [M:0;jenkins-hbase4:37181] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=344.81 KB at sequenceid=578 (bloomFilter=true), to=hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/15a5865793ad4eb39c561b4cca7afbb2 2023-04-19 20:56:32,241 INFO [M:0;jenkins-hbase4:37181] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 15a5865793ad4eb39c561b4cca7afbb2 2023-04-19 20:56:32,241 DEBUG [M:0;jenkins-hbase4:37181] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/15a5865793ad4eb39c561b4cca7afbb2 as hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/15a5865793ad4eb39c561b4cca7afbb2 2023-04-19 20:56:32,244 INFO [M:0;jenkins-hbase4:37181] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 15a5865793ad4eb39c561b4cca7afbb2 2023-04-19 20:56:32,244 INFO [M:0;jenkins-hbase4:37181] regionserver.HStore(1080): Added hdfs://localhost:43431/user/jenkins/test-data/441de9ac-1eb3-c8f5-018c-ebf4d9135e49/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/15a5865793ad4eb39c561b4cca7afbb2, entries=65, sequenceid=578, filesize=24.6 K 2023-04-19 20:56:32,245 INFO [M:0;jenkins-hbase4:37181] regionserver.HRegion(2948): Finished flush of dataSize ~344.81 KB/353082, heapSize ~393.59 KB/403032, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 20ms, sequenceid=578, compaction requested=false 2023-04-19 20:56:32,246 INFO [M:0;jenkins-hbase4:37181] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-19 20:56:32,246 DEBUG [M:0;jenkins-hbase4:37181] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-04-19 20:56:32,248 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-04-19 20:56:32,248 INFO [M:0;jenkins-hbase4:37181] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-04-19 20:56:32,249 INFO [M:0;jenkins-hbase4:37181] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:37181 2023-04-19 20:56:32,252 DEBUG [M:0;jenkins-hbase4:37181] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase4.apache.org,37181,1681937615733 already deleted, retry=false 2023-04-19 20:56:32,320 INFO [RS:0;jenkins-hbase4:35247] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,35247,1681937616971; zookeeper connection closed. 2023-04-19 20:56:32,320 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-19 20:56:32,320 ERROR [Listener at localhost/34893-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@3b1af435 rejected from java.util.concurrent.ThreadPoolExecutor@2469eb06[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2461] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-19 20:56:32,320 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@35dc457] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@35dc457 2023-04-19 20:56:32,321 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): regionserver:35247-0x1019e3331ea0001, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-19 20:56:32,321 ERROR [Listener at localhost/34893-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@1c7ec6e3 rejected from java.util.concurrent.ThreadPoolExecutor@2469eb06[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2461] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-19 20:56:32,322 INFO [Listener at localhost/34893] util.JVMClusterUtil(335): Shutdown of 1 master(s) and 1 regionserver(s) complete 2023-04-19 20:56:32,420 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-19 20:56:32,420 INFO [M:0;jenkins-hbase4:37181] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,37181,1681937615733; zookeeper connection closed. 2023-04-19 20:56:32,420 ERROR [Listener at localhost/34893-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@d12e856 rejected from java.util.concurrent.ThreadPoolExecutor@468a2451[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4121] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-19 20:56:32,421 DEBUG [Listener at localhost/34893-EventThread] zookeeper.ZKWatcher(600): master:37181-0x1019e3331ea0000, quorum=127.0.0.1:58736, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-19 20:56:32,421 ERROR [Listener at localhost/34893-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@50fbc2b9 rejected from java.util.concurrent.ThreadPoolExecutor@468a2451[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4121] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-19 20:56:32,422 WARN [Listener at localhost/34893] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-04-19 20:56:32,434 INFO [Listener at localhost/34893] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-04-19 20:56:32,542 WARN [BP-577765149-172.31.14.131-1681937612442 heartbeating to localhost/127.0.0.1:43431] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-04-19 20:56:32,542 WARN [BP-577765149-172.31.14.131-1681937612442 heartbeating to localhost/127.0.0.1:43431] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-577765149-172.31.14.131-1681937612442 (Datanode Uuid db110760-0b31-424d-81dc-8e77f2c753f3) service to localhost/127.0.0.1:43431 2023-04-19 20:56:32,545 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b3568cac-f3a4-dbc0-683e-f8685e6f55fc/cluster_dc7aee7d-e9b8-c554-2b32-38e4cf91128a/dfs/data/data1/current/BP-577765149-172.31.14.131-1681937612442] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-04-19 20:56:32,546 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b3568cac-f3a4-dbc0-683e-f8685e6f55fc/cluster_dc7aee7d-e9b8-c554-2b32-38e4cf91128a/dfs/data/data2/current/BP-577765149-172.31.14.131-1681937612442] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-04-19 20:56:32,586 INFO [Listener at localhost/34893] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-04-19 20:56:32,760 INFO [Listener at localhost/34893] zookeeper.MiniZooKeeperCluster(344): Shutdown MiniZK cluster with all ZK servers 2023-04-19 20:56:32,962 INFO [Listener at localhost/34893] hbase.HBaseTestingUtility(1293): Minicluster is down